Reproducible Results Policy

Rosenberg, David E.; Jones, Amber Spackman; Filion, Yves; Teasley, Rebecca; Sandoval-Solís, Samuel; Stagge, James H.; Abdallah, Adel M.; Castronova, Anthony M.; Ostfeld, Avi; Watkins, David

doi:10.1061/(asce)wr.1943-5452.0001368

Cited by 10 publications

(4 citation statements)

References 3 publications

(7 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The allegorical "stick" of higher reproducibility standards should be accompanied by carrots in the form of publication incentives. For example, the Journal of Water Resources Planning and Management waives or reduces fees of reproducible papers, 227 and several journals promote reproducible works, e.g., through badges. 228 Other carrots could involve a support mechanism by which pre- and post-publication peer review can properly evaluate and test statistical methods cited in papers.…”

Section: Discussionmentioning

confidence: 99%

The role of metadata in reproducible computational research

et al. 2021

View full text Add to dashboard Cite

Summary Reproducible computational research (RCR) is the keystone of the scientific method for in silico analyses, packaging the transformation of raw data to published results. In addition to its role in research integrity, improving the reproducibility of scientific studies can accelerate evaluation and reuse. This potential and wide support for the FAIR principles have motivated interest in metadata standards supporting reproducibility. Metadata provide context and provenance to raw data and methods and are essential to both discovery and validation. Despite this shared connection with scientific data, few studies have explicitly described how metadata enable reproducible computational research. This review employs a functional content analysis to identify metadata standards that support reproducibility across an analytic stack consisting of input data, tools, notebooks, pipelines, and publications. Our review provides background context, explores gaps, and discovers component trends of embeddedness and methodology weight from which we derive recommendations for future work.

show abstract

Section: Discussionmentioning

confidence: 99%

The role of metadata in reproducible computational research

et al. 2021

View full text Add to dashboard Cite

show abstract

“…The journal Cortex has a special article type Verification Reports , which are actually about replication of results and are very well designed/reasoned 83 . The Journal of Water Resources Planning and Management’s policy recognises reproducible papers in a special collection and incentivises authors with waived or reduced fees 84 . In a similar vein, the CODECHECK certificates could also be published as a special article type within journals.…”

Section: Related Workmentioning

confidence: 99%

CODECHECK: an Open Science initiative for the independent execution of computations underlying research articles during peer review to improve reproducibility

Nüst

Eglen

2021

F1000Res

View full text Add to dashboard Cite

The traditional scientific paper falls short of effectively communicating computational research. To help improve this situation, we propose a system by which the computational workflows underlying research articles are checked. The CODECHECK system uses open infrastructure and tools and can be integrated into review and publication processes in multiple ways. We describe these integrations along multiple dimensions (importance, who, openness, when). In collaboration with academic publishers and conferences, we demonstrate CODECHECK with 25 reproductions of diverse scientific publications. These CODECHECKs show that asking for reproducible workflows during a collaborative review can effectively improve executability. While CODECHECK has clear limitations, it may represent a building block in Open Science and publishing ecosystems for improving the reproducibility, appreciation, and, potentially, the quality of non-textual research artefacts. The CODECHECK website can be accessed here: https://codecheck.org.uk/.

show abstract

“…A select number of research dissemination outlets in the water treatment field have recently strengthened their commitments to FAIR/O data norms as part of a broader shift toward FAIR/O principles. For example, JWRPM ’s new reproducible results policy involves associate editors reviewing author’s submissions for reproducibility and (provisionally) incentivizing researchers meeting this higher standard with free open access publication . Editors could further support our field’s progress toward FAIR/O data ideals by defining a framework for accountability so that data sharing requirements are consistently enforced.…”

Section: Actionable Steps Toward a Fair/o Data Hub For Water Treatmentmentioning

confidence: 99%

“…For example, JWRPM's new reproducible results policy involves associate editors reviewing author's submissions for reproducibility and (provisionally) incentivizing researchers meeting this higher standard with free open access publication. 76 Editors could further support our field's progress toward FAIR/O data ideals by defining a framework for accountability so that data sharing requirements are consistently enforced. However, motivating these changes may be difficult for editors when it lies outside their current work scope and defies our field's present research culture.…”

Section: ■ Introductionmentioning

confidence: 99%

Recommendations for Advancing FAIR and Open Data Standards in the Water Treatment Community

2021

View full text Add to dashboard Cite

Water treatment researchers will maximize insights from data by implementing FAIR and Open (FAIR/O) principles for data storage and management. Progress toward domain-specific FAIR/O data infrastructure and practices advance data synthesizability, data extensibility, and stakeholder diversity. Embracing FAIR/O data principles in domain research would enable data-driven research breakthroughs that benefit a diverse array of water treatment research stakeholders. These breakthroughs arise from increased data density to improve research scope, better cross-mode data translation between experimental and modeling research outputs, and improved multiscale data integration to answer key questions across research scales. This paper addresses the gap between the current state of data sharing and the envisioned potential of FAIR/O data by offering practical steps toward building effective and sustainable water treatment data infrastructure, engaging key stakeholders, and motivating their participation. We clarify essential activities of a water treatment research data repository and suggest opportunities for communicating and growing its value.

show abstract

Reproducible Results Policy

Cited by 10 publications

References 3 publications

The role of metadata in reproducible computational research

The role of metadata in reproducible computational research

CODECHECK: an Open Science initiative for the independent execution of computations underlying research articles during peer review to improve reproducibility

Recommendations for Advancing FAIR and Open Data Standards in the Water Treatment Community

Contact Info

Product

Resources

About