Computational reproducibility of scientific workflows at extreme scales

Pouchard, Line; Baldwin, Sterling; Elsethagen, Todd; Jha, Shantenu; Raju, Bibi; Stephan, Eric G.; Tang, Liqiong; Dam, Kerstin Kleese van

doi:10.1177/1094342019839124

Cited by 14 publications

(7 citation statements)

References 43 publications

(39 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…These types of programmes are capable of building all necessary software in a managed way. Workflow management systems can capture detailed experiment provenance information for reproducibility and allow scientists to define the workflow of their computational experiments more robustly [89][90][91][92][93][94][95][96]. How to execute computational experiments is made clearer when using such systems.…”

Section: (A) Relationship To Prior Workmentioning

confidence: 99%

Learning from reproducing computational results: introducing three principles and theReproduction Package

Krafczyk

Shi

Bhaskar

et al. 2021

Phil. Trans. R. Soc. A.

View full text Add to dashboard Cite

We carry out efforts to reproduce computational results for seven published articles and identify barriers to computational reproducibility. We then derive three principles to guide the practice and dissemination of reproducible computational research: (i) Provide transparency regarding how computational results are produced; (ii) When writing and releasing research software, aim for ease of (re-)executability; (iii) Make any code upon which the results rely as deterministic as possible. We then exemplify these three principles with 12 specific guidelines for their implementation in practice. We illustrate the three principles of reproducible research with a series of vignettes from our experimental reproducibility work. We define a novel Reproduction Package , a formalism that specifies a structured way to share computational research artifacts that implements the guidelines generated from our reproduction efforts to allow others to build, reproduce and extend computational science. We make our reproduction efforts in this paper publicly available as exemplar Reproduction Packages . This article is part of the theme issue ‘Reliability and reproducibility in computational science: implementing verification, validation and uncertainty quantification in silico ’.

show abstract

Section: (A) Relationship To Prior Workmentioning

confidence: 99%

Learning from reproducing computational results: introducing three principles and theReproduction Package

Krafczyk

Shi

Bhaskar

et al. 2021

Phil. Trans. R. Soc. A.

View full text Add to dashboard Cite

show abstract

“…Global summations run in parallel may be non-deterministic; larger problems are likely to lead to lower reproducibility due to additional threads and processors. Unfortunately, there are no community standards for acceptable reproducibility on exascale systems (Pouchard et al, 2019). Our implementation offers an option to improve reproducibility with a minimal impact to performance, allowing designers to better control the acceptable amount of reproducibility to performance ratios.…”

Section: Reproducibilitymentioning

confidence: 99%

“…Variables prefixed with l are the local terms. 512-bit vector would use vector (Pouchard et al, 2019), 8*double, and the for loop would increment by 8 for each iteration. The implementation is limited to where there is a Cþþ compiler.…”

Section: Kahan Algorithm Vectorizedmentioning

confidence: 99%

Fast, good, and repeatable: Summations, vectorization, and reproducibility

Neuman

DuBois

Monroe

et al. 2020

The International Journal of High Performance Computing Applica

View full text Add to dashboard Cite

Enhanced-precision global sums are key to reproducibility in exascale applications. We examine two classic summation algorithms and show that vectorized versions are fast, good and reproducible at exascale. Both 256-bit and 512-bit implementations speed up the operation by almost a factor of four over the serial version. They thus demonstrate improved performance on global summations while retaining the numerical reproducibility of these methods.

show abstract

“…Reporting data provenance is crucial for building trust and traceability of data. By including metadata that describe the data production process, comprehensive documentation guarantees the preservation of data and methodologies, ensures repeatability, and supports analysis. − …”

Section: Introductionmentioning

confidence: 99%

“…The need for usable, automated workflows has been recognized in many research fields, including MM. ,,− ,,− Community standards are needed on how MM data should be reported and exchanged. Each data-generating process must be included as metadata to enable repeatability, replicability, and reproducibility to guarantee that users can reliably repeat a process under identical conditions.…”

Section: Introductionmentioning

confidence: 99%

Simulation Foundry: Automated and F.A.I.R. Molecular Modeling

Gygli

Pleiss

2020

J. Chem. Inf. Model.

View full text Add to dashboard Cite

The Simulation Foundry (SF) is a modular workflow for the automated creation of molecular modeling (MM) data. MM allows for the reliable prediction of the microscopic and macroscopic properties of multicomponent systems from first principles. The SF makes MM repeatable, replicable, and findable, accessible, interoperable, and reusable (F.A.I.R.). The SF uses a standardized data structure and file naming convention, allowing for replication on different supercomputers and re-entrancy. We focus on keeping the SF simple by basing it on scripting languages that are widely used by the MM community (bash, Python) and making it reusable and re-editable. The SF was developed to assist expert users in performing parameter studies of multicomponent systems by high throughput molecular dynamics simulations. The usability of the SF is demonstrated by simulations of thermophysical properties of binary mixtures. A standardized data exchange format enables the integration of simulated data with data from experiments. The SF also provides a complete documentation of how the results were obtained, thus assigning provenance. Increasing computational power facilitates the intensification of the simulation process and requires automation and modularity. The SF provides a community platform on which to integrate new methods and create data that is reproducible and transparent (, ).

show abstract

Computational reproducibility of scientific workflows at extreme scales

Cited by 14 publications

References 43 publications

Learning from reproducing computational results: introducing three principles and theReproduction Package

Learning from reproducing computational results: introducing three principles and theReproduction Package

Fast, good, and repeatable: Summations, vectorization, and reproducibility

Simulation Foundry: Automated and F.A.I.R. Molecular Modeling

Contact Info

Product

Resources

About