An automated and reproducible workflow for running and analyzing neural simulations using Lancet and IPython Notebook

Abstract

Lancet is a new, simulator-independent Python utility for succinctly specifying, launching, and collating results from large batches of interrelated computationally demanding program runs. This paper demonstrates how to combine Lancet with IPython Notebook to provide a flexible, lightweight, and agile workflow for fully reproducible scientific research. This informal and pragmatic approach uses IPython Notebook to capture the steps in a scientific computation as it is gradually automated and made ready for publication, without mandating the use of any separate application that can constrain scientific exploration and innovation. The resulting notebook concisely records each step involved in even very complex computational processes that led to a particular figure or numerical result, allowing the complete chain of events to be replicated automatically. Lancet was originally designed to help solve problems in computational neuroscience, such as analyzing the sensitivity of a complex simulation to various parameters, or collecting the results from multiple runs with different random starting points. However, because it is never possible to know in advance what tools might be required in future tasks, Lancet has been designed to be completely general, supporting any type of program as long as it can be launched as a process and can return output in the form of files. For instance, Lancet is also heavily used by one of the authors in a separate research group for launching batches of microprocessor simulations. This general design will allow Lancet to continue supporting a given research project even as the underlying approaches and tools change.

Reproducibility in Computational Neuroscience Models and Simulations

“…Internal reproducibility can be further enhanced by adherence to standard workflows and by implementing these as much as possible through workflow automation with tools like Mozaik [155] and Lancet [156]. Similarly, databases associated with parameter provenance and other research documents can also be added to one or more repositories associated with a research project.…”

automan: A Python-Based Automation Framework for Numerical Computing

“…• Lancet [13] provides general abstractions to decompose running a suite of programs into a parametrization of options, specification of the actual commands to run, and how to actually execute the task. Unfortunately, this does not include any task management or automation of tasks and their dependencies.…”

A Survey on Collecting, Managing, and Analyzing Provenance from Scripts

“…However, database systems work better for provenance comprehension and for storing other types of provenance due to the possibility of querying and the capability of storing non-file artifacts, such as function calls, variables, and environment variables [62,98]. Although file systems are also viable for such non-file data, they require the provenance tools to implement their own serialization mechanisms [54,103,121].…”