Replicability of Research in Biomedical Natural Language Processing:
            a pilot evaluation for a coding task

Névéol, Aurélie; Cohen, K. Bretonnel; Grouin, Cyril; Robert, Aude

doi:10.18653/v1/w16-6110

Cited by 4 publications

(2 citation statements)

References 11 publications

(4 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…must exist along with published papers. As part of a NLP challenge, Névéol et al (2016) report results on replicating experiments from three systems submitted to the CLEF eHealth track. They show that replication is feasible although "ease of replicating results varied".…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

NLP Community Perspectives on Replicability

Mieskes¹,

Fort

Névéol

et al. 2019

Proceedings - Natural Language Processing in a Deep Learning World

Self Cite

View full text Add to dashboard Cite

With recent efforts in drawing attention to the task of replicating and/or reproducing 1 results, for example in the context of COLING 2018 and various LREC workshops, the question arises how the NLP community views the topic of replicability in general. Using a survey, in which we involve members of the NLP community, we investigate how our community perceives this topic, its relevance and options for improvement. Based on over two hundred participants, the survey results confirm earlier observations, that successful reproducibility requires more than having access to code and data. Additionally, the results show that the topic has to be tackled from the authors', reviewers' and community's side.

show abstract

Section: Related Workmentioning

confidence: 99%

“…The next steps include, but are not limited to, analyzing whether the supplementary material and appendices actually do improve replicability, as stated by Névéol et al (2016). Furthermore, evaluating the repository offered by Fares et al (2017), whether other researchers actually build on top of it and with what results.…”

Section: Future Workmentioning

confidence: 99%

NLP Community Perspectives on Replicability

Mieskes¹,

Fort

Névéol

et al. 2019

Proceedings - Natural Language Processing in a Deep Learning World

Self Cite

View full text Add to dashboard Cite

show abstract

The role of machine learning in developing non-magnetic resonance imaging based biomarkers for multiple sclerosis: a systematic review

Hossain

Daskalaki

Brüstle

et al. 2022

BMC Med Inform Decis Mak

View full text Add to dashboard Cite

Background Multiple sclerosis (MS) is a neurological condition whose symptoms, severity, and progression over time vary enormously among individuals. Ideally, each person living with MS should be provided with an accurate prognosis at the time of diagnosis, precision in initial and subsequent treatment decisions, and improved timeliness in detecting the need to reassess treatment regimens. To manage these three components, discovering an accurate, objective measure of overall disease severity is essential. Machine learning (ML) algorithms can contribute to finding such a clinically useful biomarker of MS through their ability to search and analyze datasets about potential biomarkers at scale. Our aim was to conduct a systematic review to determine how, and in what way, ML has been applied to the study of MS biomarkers on data from sources other than magnetic resonance imaging. Methods Systematic searches through eight databases were conducted for literature published in 2014–2020 on MS and specified ML algorithms. Results Of the 1, 052 returned papers, 66 met the inclusion criteria. All included papers addressed developing classifiers for MS identification or measuring its progression, typically, using hold-out evaluation on subsets of fewer than 200 participants with MS. These classifiers focused on biomarkers of MS, ranging from those derived from omics and phenotypical data (34.5% clinical, 33.3% biological, 23.0% physiological, and 9.2% drug response). Algorithmic choices were dependent on both the amount of data available for supervised ML (91.5%; 49.2% classification and 42.3% regression) and the requirement to be able to justify the resulting decision-making principles in healthcare settings. Therefore, algorithms based on decision trees and support vector machines were commonly used, and the maximum average performance of 89.9% AUC was found in random forests comparing with other ML algorithms. Conclusions ML is applicable to determining how candidate biomarkers perform in the assessment of disease severity. However, applying ML research to develop decision aids to help clinicians optimize treatment strategies and analyze treatment responses in individual patients calls for creating appropriate data resources and shared experimental protocols. They should target proceeding from segregated classification of signals or natural language to both holistic analyses across data modalities and clinically-meaningful differentiation of disease. Supplementary Information The online version contains supplementary material available at 10.1186/s12911-022-01985-5.

show abstract

Bayes Test of Precision, Recall, and F1 Measure for Comparison of Two Natural Language Processing Models

Wang¹,

Li²

2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

Direct comparison on point estimation of the precision (P), recall (R), and F 1 measure of two natural language processing (NLP) models on a common test corpus is unreasonable and results in less replicable conclusions due to a lack of a statistical test. However, the existing t-tests in cross-validation (CV) for model comparison are inappropriate because the distributions of P, R, F 1 are skewed and an interval estimation of P, R, and F 1 based on a t-test may exceed [0,1]. In this study, we propose to use a block-regularized 3 × 2 CV (3 × 2 BCV) in model comparison because it could regularize the difference in certain frequency distributions over linguistic units between training and validation sets and yield stable estimators of P, R, and F 1. On the basis of the 3 × 2 BCV, we calibrate the posterior distributions of P, R, and F 1 and derive an accurate interval estimation of P, R, and F 1. Furthermore, we formulate the comparison into a hypothesis testing problem and propose a novel Bayes test. The test could directly compute the probabilities of the hypotheses on the basis of the posterior distributions and provide more informative decisions than the existing significance t-tests. Three experiments with regard to NLP chunking tasks are conducted, and the results illustrate the validity of the Bayes test.

show abstract

Replicability of Research in Biomedical Natural Language Processing: a pilot evaluation for a coding task

Cited by 4 publications

References 11 publications

NLP Community Perspectives on Replicability

NLP Community Perspectives on Replicability

The role of machine learning in developing non-magnetic resonance imaging based biomarkers for multiple sclerosis: a systematic review

Bayes Test of Precision, Recall, and F1 Measure for Comparison of Two Natural Language Processing Models

Contact Info

Product

Resources

About