Most computational hydrology is not reproducible, so is it really science?

Abstract

Reproducibility is a foundational principle in scientific research. Yet in computational hydrology the code and data that actually produces published results are not regularly made available, inhibiting the ability of the community to reproduce and verify previous findings. In order to overcome this problem we recommend that reuseable code and formal workflows, which unambiguously reproduce published scientific results, are made available for the community alongside data, so that we can verify previous findings, and build directly from previous work. In cases where reproducing large‐scale hydrologic studies is computationally very expensive and time‐consuming, new processes are required to ensure scientific rigor. Such changes will strongly improve the transparency of hydrological research, and thus provide a more credible foundation for scientific advancement and policy support.

Reply to comment by Añel on “Most computational hydrology is not reproducible, so is it really science?”

“…We fully agree that the problematic issue with reproducibility is not new, neither to science nor to computing technology, and we explicitly acknowledge this in the commentary as problem more broadly in the scientific literature [Hutton et al, 2016]. However, for too long it has been neglected in our scientific community of hydrology.…”

“…However, in our view, it is the cultural change that is the greatest challenge to overcome to achieve reproducible scientific research in computational hydrology. We believe that from changing the culture and attitude among hydrological scientists, details will evolve to cover more (technical) aspects over time.We would first like to thank Añel [2017] for responding to our commentary paper [Hutton et al, 2016], and contributing some important additional points to the reproducibility issues we have raised. We fully agree that the problematic issue with reproducibility is not new, neither to science nor to computing technology, and we explicitly acknowledge this in the commentary as problem more broadly in the scientific literature [Hutton et al, 2016].…”

“…We would first like to thank Añel [2017] for responding to our commentary paper [Hutton et al, 2016], and contributing some important additional points to the reproducibility issues we have raised. We fully agree that the problematic issue with reproducibility is not new, neither to science nor to computing technology, and we explicitly acknowledge this in the commentary as problem more broadly in the scientific literature [Hutton et al, 2016].…”

“…Arguably more than in areas such as physiology and pharmacology, where the reproducibility issue is already well discussed (see citations in Hutton et al [2016]). We argue that how the scientist and modeler deals with those boundary conditions must be documented in the workflow of experiments.…”

“…These will likely be included as metadata fields for code and workflows, as we discuss the need for in our commentary [Hutton et al, 2016], which need to be evolved for hydrological code. Using larger model setups requires not only version management of the code but also requires version management of the full setup, including input data and how it was treated.…”