Enhancing reproducibility for computational methods

Stodden, Victoria; McNutt, Marcia; Bailey, David H.; Deelman, Ewa; Gil, Yolanda; Hanson, Brooks; Heroux, Michael A.; Ioannidis, John P. A.; Taufer, Michela

doi:10.1126/science.aah6168

Cited by 308 publications

(306 citation statements)

References 10 publications

Supporting

Mentioning

300

Contrasting

Order By: Relevance

“…Below we summarize some further recommendations which we think can minimize the negative features of NHST even if it continues to be dominant for a while. A very important practical goal would be to change the incentive structure of biomedical and social science to bring it in line with these and similar other recommendations (Wagenmakers et al, 2011;Begley and Ellis, 2012;Nosek et al, 2013;Stodden et al, 2016). Also note that we are not arguing against statistical inference which we consider the "logic of science" (Jaynes, 2003;p.…”

Section: Ways To Changementioning

confidence: 85%

See 1 more Smart Citation

When null hypothesis significance testing is unsuitable for research: a reassessment

Szűcs

Ioannidis

2016

Preprint

107

View full text Add to dashboard Cite

Null hypothesis significance testing (NHST) has several shortcomings that are likely contributing factors behind the widely debated replication crisis of (cognitive) neuroscience, psychology, and biomedical science in general. We review these shortcomings and suggest that, after sustained negative experience, NHST should no longer be the default, dominant statistical practice of all biomedical and psychological research. If theoretical predictions are weak we should not rely on all or nothing hypothesis tests. Different inferential methods may be most suitable for different types of research questions. Whenever researchers use NHST they should justify its use, and publish pre-study power calculations and effect sizes, including negative findings. Hypothesis-testing studies should be pre-registered and optimally raw data published. The current statistics lite educational approach for students that has sustained the widespread, spurious use of NHST should be phased out.

show abstract

Section: Ways To Changementioning

confidence: 85%

“…Perhaps specialized methods repository journals could be set up for this purpose. For some recent recommendations for improving computational reproducibility practices see Stodden et al (2016).…”

Section: Publish All Analysis Scripts With Analysis Settingsmentioning

confidence: 99%

When null hypothesis significance testing is unsuitable for research: a reassessment

Szűcs

Ioannidis

2016

Preprint

107

View full text Add to dashboard Cite

show abstract

“…This was 5 developed in accordance with state-of-the-art data lifecycle management and to maintain interoperability (Stodden et al, 2016).…”

Section: Provenance Trackingmentioning

confidence: 99%

“…Recently, Stodden et al (2016) presented "Reproducibility Enhancement Principles" (REP) to help ensure that the computational steps in data processing and generation are similarly important as access to the 25 data themselves. Hence sharing not only data but also details on software, workflows, and the computational environment via open repositories is likewise important.…”

mentioning

confidence: 99%

Best practice regarding the three P's: profiling, portability and provenance when running HPC geoscientific applications

Sharples

Zhukov

Geimer

et al. 2017

Preprint

View full text Add to dashboard Cite

Abstract. Geoscientific modeling is constantly evolving, with next generation geoscientific models and applications placing high demands on high performance computing (HPC) resources. These demands are being met by new developments in HPC architectures, software libraries, and infrastructures. New HPC developments require new programming paradigms leading to substantial investment in model porting, tuning, and refactoring of complicated legacy code in order to use these resources effectively. In addition to the challenge of new massively parallel HPC systems, reproducibility of simulation and analysis 5 results is of great concern, as the next generation geoscientific models are based on complex model implementations and profiling, modeling and data processing workflows.Thus, in order to reduce both the duration and the cost of code migration, aid in the development of new models or model components, while ensuring reproducibility and sustainability over the complete data life cycle, a streamlined approach to profiling, porting, and provenance tracking is necessary. We propose a run control framework (RCF) integrated with a workflow 10 engine which encompasses all stages of the modeling chain: 1. preprocess input, 2. compilation of code (including code instrumentation with performance analysis tools), 3. simulation run, 4. postprocess and analysis, to address these issues. Within this RCF, the workflow engine is used to create and manage benchmark or simulation parameter combinations and performs the documentation and data organization for reproducibility. This approach automates the process of porting and tuning, profiling, testing, and running a geoscientific model. We show that in using our run control framework, testing, benchmarking, profiling, 15 and running models is less time consuming and more robust, resulting in more efficient use of HPC resources, more strategic code development, and enhanced data integrity and reproducibility.

show abstract

“…As data become more open and transportable, a second layer of research output, linking publications to the associated data, has emerged [2]. This is now followed by the recognition of an important and new third layer: communicating the process of inquiry itself, i.e., a complete computational narrative, through the linking and sharing of methods, source code, and data, thereby introducing a new model of reproducible science and accelerated knowledge discovery [13]. The Whole Tale strengthens the second layer (linking data, code, and digital scholarly objects to publications) and also builds a robust third layer that integrates all parts of the research story into a computational tale (conveying the holistic experience of reproducible scientific inquiry, i.e., sharing the source code, data, and methods, along with the computational environment in which inquiry is conducted) and making both layers accessible from the scholarly publication.…”

Section: Introductionmentioning

confidence: 99%