Computing environments for reproducibility: Capturing the “Whole Tale”

Brinckman, Adam; Chard, Kyle; Gaffney, Niall; Hategan, Mihael; Jones, Matthew B.; Kowalik, Kacper; Kulasekaran, Sivakumar; Ludäscher, Bertram; Mecum, Bryce; Nabrzyski, Jarek; Stodden, Victoria; Taylor, Ian; Turk, Matthew J.; Turner, Kandace

doi:10.1016/j.future.2017.12.029

Cited by 108 publications

(75 citation statements)

References 54 publications

(58 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Nonetheless, with the introduction of computers there has been a gradual and unintended drift away from reproducibility (Bailey Borwein & Stodden 2016;Brinckman et al 2019), and so from the fundamental tenets of scientific practice. Many archaeological tasks have shifted from the descriptive and qualitative aspects to quantitative and computational analyses.…”

Section: Discussionmentioning

confidence: 99%

Tool-Driven Revolutions in Archaeological Science

Schmidt

Marwick

2020

Journal of Computer Applications in Archaeology

View full text Add to dashboard Cite

IntroductionIn this paper we investigate recent developments which are facilitating computational archaeology's return to fundamental principles of the scientific method. We ask whether this might be the beginning of a broader change in the field. Revolutions or paradigm-shifts in archaeology have been proposed to occur following major theoretical statements (Clark 1993;Härke 2002), but there is an argument among philosophers of science whether revolutions in science are idea-driven or tool-driven (Dyson 2000;Galison 1997). This argument motivated us to explore how the disciplines of ecology and archaeology have started to embrace tools that improve the reproducibility of research. We focus on one set of tools for this change, based on writing data analysis scripts in free and open source programming languages, exemplified by R, and the practice of sharing of these scripts. We focus on R because our prior observations indicate it is by far the most common scripting language used by ecologists and archaeologists. Bibliometric evidence shows a strong increase in the use of R among ecologists and the beginning of a similar development in archaeology. We evaluate how approaches to improving reproducibility and transparency in archaeology are mediated and transformed by digital approaches and propose these might reflect a tool-driven change in archaeology. Acknowledging that this is not a simple process, we offer an R-based tool to ease the task of creating a compen-dium which enables other researchers to reproduce the published results.

show abstract

Section: Discussionmentioning

confidence: 99%

Tool-Driven Revolutions in Archaeological Science

Schmidt

Marwick

2020

Journal of Computer Applications in Archaeology

View full text Add to dashboard Cite

show abstract

“…Reproducibility: Concerns about reproducibility are having a profound effect on research [17]. While reproducibility initiatives have primarily focused on making data and experimental processes available to reproduce findings, there is a growing interest in making computational methods available as well [18][19][20].…”

Section: Specialized Requirements Of Sciencementioning

confidence: 99%

DLHub: Model and Data Serving for Science

Chard

et al. 2019

2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS)

Self Cite

View full text Add to dashboard Cite

While the Machine Learning (ML) landscape is evolving rapidly, there has been a relative lag in the development of the "learning systems" needed to enable broad adoption. Furthermore, few such systems are designed to support the specialized requirements of scientific ML. Here we present the Data and Learning Hub for science (DLHub), a multi-tenant system that provides both model repository and serving capabilities with a focus on science applications. DLHub addresses two significant shortcomings in current systems. First, its selfservice model repository allows users to share, publish, verify, reproduce, and reuse models, and addresses concerns related to model reproducibility by packaging and distributing models and all constituent components. Second, it implements scalable and low-latency serving capabilities that can leverage parallel and distributed computing resources to democratize access to published models through a simple web interface. Unlike other model serving frameworks, DLHub can store and serve any Python 3compatible model or processing function, plus multiple-function pipelines. We show that relative to other model serving systems including TensorFlow Serving, SageMaker, and Clipper, DLHub provides greater capabilities, comparable performance without memoization and batching, and significantly better performance when the latter two techniques can be employed. We also describe early uses of DLHub for scientific applications.

show abstract

“…Simflowny ( Arbona et al, 2013) is a platform for formalizing and managing the main elements of a simulation flow, tied not to a field, but to a specified simulation architecture. The Whole Tale approach (Brinckman et al, 2018) aims at linking data, code, and digital scholarly objects to publications and integrating all parts of the research story. Other works focus on code and data management, such as Ram (2013) suggesting very general version control systems such as Git for transparent data management in order to enable reproducibility of scientific work.…”

Section: Related Workmentioning

confidence: 99%

Making simulation results reproducible—Survey, guidelines, and examples based on Gradle and Docker

Elmenreich

Moll

Theuermann

et al. 2019

PeerJ Computer Science

View full text Add to dashboard Cite

This article addresses two research questions related to reproducibility within the context of research related to computer science. First, a survey on reproducibility addressed to researchers in the academic and private sectors is described and evaluated. The survey indicates a strong need for open and easily accessible results, in particular, reproducing an experiment should not require too much effort. The results of the survey are then used to formulate guidelines for making research results reproducible. In addition, this article explores four approaches based on software tools that could bring forward reproducibility in research results. After a general analysis of tools, three examples are further investigated based on actual research projects which are used to evaluate previously introduced tools. Results indicate that the evaluated tools contribute well to making simulation results reproducible but due to conflicting requirements, none of the presented solutions fulfills all intended goals perfectly.

show abstract

Computing environments for reproducibility: Capturing the “Whole Tale”

Cited by 108 publications

References 54 publications

Tool-Driven Revolutions in Archaeological Science

Tool-Driven Revolutions in Archaeological Science

DLHub: Model and Data Serving for Science

Making simulation results reproducible—Survey, guidelines, and examples based on Gradle and Docker

Contact Info

Product

Resources

About