Reproducible Computational Workflows with Continuous Analysis

Abstract

Reproducing experiments is vital to science. Being able to replicate, validate and extend previous work also speeds new research projects. Reproducing computational biology experiments, which are scripted, should be straightforward. But reproducing such work remains challenging and time consuming. In the ideal world we would be able to quickly and easily rewind to the precise computing environment where results were generated. We would then be able to reproduce the original analysis or perform new analyses. We introduce a process termed “continuous analysis” which provides inherent reproducibility to computational research at a minimal cost to the researcher. Continuous analysis combines Docker, a container service similar to virtual machines, with continuous integration, a popular software development technique, to automatically re-run computational analysis whenever relevant changes are made to the source code. This allows results to be reproduced quickly, accurately and without needing to contact the original authors. Continuous analysis also provides an audit trail for analyses that use data with sharing restrictions. This allows reviewers, editors, and readers to verify reproducibility without manually downloading and rerunning any code. Example configurations are available at our online repository (https://github.com/greenelab/continuous_analysis).

Salmon provides fast and bias-aware quantification of transcript expression

“…This allows us to track accuracy over various Salmon commits, and to identify the commit corresponding to any performance regressions. We note that this setup overlaps considerably with the setup suggested for “continuous analysis” by Beaulieu-Jones and Greene [ 32 ]. Going forward, we anticipate expanding the test suite to include even more data and performance metrics.…”

The role of metadata in reproducible computational research

“…The last two advances, exemplified by the Conda and Docker projects (described below), have largely made computational reproducibility possible, at least in the narrow sense of being able to reliably version and install software and related dependencies on other people's machines. Often small changes in software and reference data can have significant effects of an analysis 105 . Tools like Docker and Conda respectively make the computing environment and version pinning software tenable, thereby producing portable and stable environments for reproducible computational research.…”

Eleven quick tips for architecting biomedical informatics workflows with cloud computing

“…Containers play an important role in the development of distributed systems by allowing tasks to be broken up into isolated units that can be scaled by increasing the number of containers running simultaneously. Additionally, containers can be leveraged for reproducible computational analysis [ 14 ]. Importantly, cloud providers often offer integration with containers such as Docker, allowing developers to manage and scale a containerized application across a cluster of servers.…”