The Next Frontier: Making Research More Reproducible

Abstract

(NSF and IES 2018) describe reproducibility as a continuum (Fig. 1). The goal is to push work up the continuum to make data, models, code, directions, and other digital artifacts used in the research available for others to reuse (availability). Then, use shared artifacts to exactly reproduce published results (reproducibility, sometimes called bit or computational reproducibility). Finally, use artifacts with existing and new data sets to replicate findings across sites or domains (replicability). For example, the Journal of Water Resources Planning and Management policy to specify the availability of data, models, and code (Rosenberg and Watkins 2018) primarily targets availability in the reproducibility continuum. This Fig. 1. Reproducibility is a continuum.

Reproducible Results Policy

“…Authors of any article type may add an optional "Reproducible Results" section to their manuscript if all research materials listed in the "Data Availability Statement" section are publicly available in a repository or online. Next, authors must follow best practices to improve result reproducibility (Rosenberg et al 2020). One best practice is to have a colleague, student, or other person not affiliated with the study reproduce manuscript results.…”

“…See Rosenberg and Watkins (2018) for further information on the precursor step to prepare a manuscript "Data Availability Statement" section. See Rosenberg et al (2020) for practices to make results more reproducible.…”

“…This 2-day limit puts an upper bound on the time commitment requested for an AER or reproducibility reviewer to reproduce results. This limit simultaneously communicates to authors the level of organization needed for data, models, code, directions, and other artifacts so AERs, reproducibility reviewers, and Journal readers can reproduce results in a timely manner even if they are not familiar with the tools or models [see additional suggestions in Rosenberg et al (2020)]. The AER will complete a short worksheet (available in the Supplemental Materials) that assesses the accuracy of the statements in the manuscript's "Reproducible Results" section, lists the results that were reproduced, and recommends improvements to make results more reproducible.…”

“…The Journal's audience is water resource professionals and scientists who may or may not be familiar with a specific software, model, or tool used in the manuscript. Therefore, authors must share code, models, and data, and sufficiently describe and document their use so the AER, a reproducibility reviewer, and Journal readers can identify and follow the steps taken to arrive at the desired results (Rosenberg et al 2020).…”

“…The Journal recognizes that some communication may be needed between the authors and the person reproducing results. • Alternatively, share inputs to and the immediate outputs from each proprietary or computationally intensive step in the workflow (Rosenberg et al 2020).…”