Replicability of Research in Biomedical Natural Language Processing: a pilot evaluation for a coding task

Aurélie Névéol, Kevin Cohen, Cyril Grouin, Aude Robert

Abstract:

The scientific community is facing raising concerns about the reproducibility of research in many fields. To address this issue in Natural Language Processing, the CLEF eHealth 2016 lab offered a replication track together with the Clinical Information Extraction task.
Herein, we report detailed results of the replication experiments carried out with the three systems submitted to the track. While all results were ultimately replicated, we found that the systems were poorly rated by analysts on documentation aspects such as ”ease of understanding system requirements” (33%) and ”provision of information while system is running” (33%). As a result, simple steps could
be taken by system authors to increase the ease of replicability of their work, thereby increasing the ease of re-using the systems. Our experiments aim to raise the awareness of the community towards the challenges of replication and community sharing of NLP systems.