Assessing data availability and research reproducibility in hydrology and water resources

Stagge, James H.; Rosenberg, David E.; Abdallah, Adel M.; Akbar, Hadia; Attallah, Nour A.; James, R.W.

doi:10.1038/sdata.2019.30

Cited by 89 publications

(92 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Comparing our estimate of 30% empirical reproducibility, to similar findings of 38% (Vanpaemel et al, 2015), 32% (Hardwicke and Ioannidis, 2018), 26% (Wicherts et al, 2006), 19% (Vines et al, 2014; extending data lifetime appears consistently difficult to achieve, across disciplines. Our estimate of 80% analytical reproducibility is difficult to quantitatively compare against similar audits, which have found reproducibility of results to be anywhere between 83% (Andrew et al, 2015), 70% (Gilbert et al, 2012), and 1.1% (Stagge et al, 2019) of surveyed publications. This is likely because criteria for defining a successful reproduction effort, given materials, are currently ambiguous.…”

Section: Implications For Evolutionary Anthropology and Ecologymentioning

confidence: 64%

Reproducibility improves exponentially over 63 years of social learning research

Minocher

Atmaca

Bavero

et al. 2020

Preprint

View full text Add to dashboard Cite

Interest in improving reproducibility, replicability and transparency of research has increased substantially across scientific fields over the last few decades. We surveyed 560 empirical, quantitative publications published between 1955 and 2018, to estimate the rate of reproducibility for research on social learning, a large subfield of behavioural ecology. We found supporting materials were available for less than 30% of publications during this period. The availability of data declines exponentially with time since publication, with a half-life of about six years, and this “data decay rate” varies systematically with both study design and study species. Conditional on materials being available, we estimate that a reasonable researcher could expect to successfully reproduce about 80% of published results, based on our evaluating a subset of 40 publications. Taken together, this indicates an overall success rate of 24% for both acquiring materials and recovering published results, with non- reproducibility of results primarily due to unavailable, incomplete, or poorly documented data. We provide recommendations to improve the reproducibility of research on the ecology and evolution of social behaviour.

show abstract

Section: Implications For Evolutionary Anthropology and Ecologymentioning

confidence: 64%

Reproducibility improves exponentially over 63 years of social learning research

Minocher

Atmaca

Bavero

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…This person could be a new student who needs to get up to speed on the methods or someone else interested in the study or results. • Ask this colleague, student, or other person to use a reproducibility survey tool (e.g., Stagge et al 2019) to provide feedback on the repository, directions, and results that they reproduced. • If the person can reproduce results, acknowledge their effort in the manuscript's data availability or results reproducibility section.…”

Section: Verify Your Results Are Reproduciblementioning

confidence: 99%

“…Follow Good Examples 1. Adopt the practices of six articles that Stagge et al (2019) awarded badges to for full and partial reproducible results. For example, these papers • Provided all model and code in a Github (Buscombe 2017;Neuwirth 2017;Xu et al 2017), institutional (Yu et al 2017), or HydroShare (Horsburgh et al 2017) repository.…”

Section: Verify Your Results Are Reproduciblementioning

confidence: 99%

“…Research results need to be independently reproduced and validated before they are accepted as fact or theory. Across numerous fields like psychology, computer systems, and water resources, there are problems reproducing research results (Aarts et al 2015;Collberg et al 2014;Hutton et al 2016;Stagge et al 2019;Stodden et al 2018). This editorial examines the challenges to reproduce research results and suggests community practices to overcome these challenges.…”

mentioning

confidence: 99%

See 1 more Smart Citation

The Next Frontier: Making Research More Reproducible

Rosenberg

Filion

Teasley

et al. 2020

J. Water Resour. Plann. Manage.

Self Cite

View full text Add to dashboard Cite

(NSF and IES 2018) describe reproducibility as a continuum (Fig. 1). The goal is to push work up the continuum to make data, models, code, directions, and other digital artifacts used in the research available for others to reuse (availability). Then, use shared artifacts to exactly reproduce published results (reproducibility, sometimes called bit or computational reproducibility). Finally, use artifacts with existing and new data sets to replicate findings across sites or domains (replicability). For example, the Journal of Water Resources Planning and Management policy to specify the availability of data, models, and code (Rosenberg and Watkins 2018) primarily targets availability in the reproducibility continuum. This Fig. 1. Reproducibility is a continuum.

show abstract

“…To minimize the effect of this loss and meet FAIR standards, it is critical to also include detailed information about the anonymization procedure via metadata and sharing code, ideally using open-source tools integrating version control for transparency, to allow for interoperability and usability by other researchers (Bakker, 2019;Lowndes et al, 2017;Stagge et al, 2019). When possible, researchers should leave jurisdiction of sensitive data to the agencies responsible for collecting and warehousing these data; where there is no such organization, they should provide synthetic examples of the data so that others can understand and replicate the anonymization procedure.…”

Section: Sharing Private Datamentioning

confidence: 99%

Balancing Open Science and Data Privacy in the Water Sciences

Zipper¹,

Whitney²,

Deines³

et al. 2019

Preprint

View full text Add to dashboard Cite

Open science practices such as publishing data and code are transforming water science by enabling synthesis and enhancing reproducibility. However, as research increasingly bridges the physical and social science domains (e.g., socio-hydrology), there is the potential for well-meaning researchers to unintentionally violate the privacy and security of individuals or communities by sharing sensitive information. Here, we identify the contexts in which privacy violations are most likely to occur, such as working with high-resolution spatial data (e.g., from remote sensing), consumer data (e.g., from smart meters), and/or digital trace data (e.g., from social media). We also suggest practices for identifying and addressing privacy concerns at the individual, institutional, and disciplinary levels. We strongly advocate that the water science community continue moving toward open science and socio-environmental research and that progress toward these goals be rooted in open and ethical data management.

show abstract

Assessing data availability and research reproducibility in hydrology and water resources

Cited by 89 publications

References 50 publications

Reproducibility improves exponentially over 63 years of social learning research

Reproducibility improves exponentially over 63 years of social learning research

The Next Frontier: Making Research More Reproducible

Balancing Open Science and Data Privacy in the Water Sciences

Contact Info

Product

Resources

About