A literature review of methods for assessment of reproducibility in science

Abstract

Introduction: In response to the US Congress petition, the National Academies of Sciences, Engineering, and Medicine investigated the status of reproducibility and replicability in science. A piece of work is reproducible if the same results can be obtained while following the methods under the same conditions and using the same data. Unavailable data, missing code, and unclear or incomplete method descriptions are common reasons for failure to reproduce results. Objectives: The motivation behind this review is to investigate the current methods for reproducibility assessment and analyze their strengths and weaknesses so that we can determine where there is room for improvement. Methods: We followed the PRISMA 2020 standard and conducted a literature review to find the current methods to assess the reproducibility of scientific articles. We made use of three databases for our search: Web of Science, Scopus, and Engineering Village. Our criteria to find relevant articles was to look for methods, algorithms, or techniques to evaluate, assess, or predict reproducibility in science. We discarded methods that were specific to a single study, or that could not be adapted to scientific articles in general. Results: We found ten articles describing methods to evaluate reproducibility, and classified them as either a prediction market, a survey, a machine learning algorithm, or a numerical method. A prediction market requires participants to bet on the reproducibility of a study. The surveys are simple and straightforward, but their performance has not been assessed rigorously. Two types of machine learning methods have been applied: handpicked features and natural language processing. Conclusion: While the machine learning methods are promising because they can be scaled to reduce time and cost for researchers, none of the models reviewed achieved an accuracy above 75%. Given the prominence of transformer models for state-of-the-art natural language processing (NLP) tasks, we believe a transformer model can achieve better accuracy.