Reproducibility in Computing Research: An Empirical Study

Abstract

In computing, research findings are often anecdotally faulted for not being reproducible. Numerous empirical studies have analyzed the reproducibility of a variety of research. Our objective, in this study, is to quantify the current state of reproducibility of research in computing based on prior research, using three reproducibility factors-Method, Data and Experiment-to measure three different degrees of reproducibility. Twenty-five variables traditionally utilized to document reproducibility are identified and grouped into three factors, namely Method, Data and Experiment. These variables describe the extent to which these factors are documented for each paper. Approximately 100 randomly selected research papers from the International Conference on Information Systems series, for the year 2019, are surveyed. Our findings suggest that none of the papers documented all the variables. In fact, the results show that relatively few variables for each factor are documented. Some of the variables vary across different categories of papers, and most papers fail in at least one of the factors. Reproducibility scores decrease with increased documentation requirements. Reproducibility may improve over time, as researchers prioritize reproducibility and utilize methods that ensure reproducibility. Research documentation in computing is remarkably limited, resulting in a dearth of reproducible factors. Future research may study the shifts and trends in reproducibility over time. Meanwhile, researchers and publishers must increase their focus on the reproducibility aspects of their papers. This study contributes to our understanding of the status quo of reproducibility in computing research.

A literature review of methods for assessment of reproducibility in science

“…It must be noted that one of the search results, Raghupathi, Raghupathi, and Ren (2022), adopts the method from Gundersen and Kjensmo (2018). These are grouped together as one method under the section 3.4.…”

“Paper, Meet Code”: A Deep Learning Approach to Linking Scholarly Articles With GitHub Repositories

“…The official relationship indicates that the authors of the corresponding paper publish the GitHub repository, often to distribute the source code used in their published research. The impact of the official relationship extends to individuals seeking to reference authoritative works, as official GitHub repositories often offer more information, reproducibility, reliability, and professionalism [5,6,7]. Previously, the traditional way to determine the relationship between a paper and its corresponding GitHub repository was to painstakingly examine the content of both.…”