NeuronUnit: A package for data-driven validation of neuron models using SciUnit

Abstract

Not peer-reviewed ABSTRACTValidating a quantitative scientific model requires comparing its predictions against many experimental observations, ideally from many labs, using transparent, robust, statistical comparisons. Unfortunately, in rapidly-growing fields like neuroscience, this is becoming increasingly untenable, even for the most conscientious scientists. Thus the merits and limitations of existing models, or whether a new model is an improvement on the state-of-the-art, is often unclear.Software engineers seeking to verify, validate and contribute to a complex software project rely on suites of simple executable tests, called "unit tests". Drawing inspiration from this practice, we previously developed SciUnit, an easy-to-use framework for developing data-driven "model validation tests" -executable functions, here written in Python. Each such test generates and statistically validates predictions from a model against one relevant feature of empirical data to produce a score indicating agreement between the model and the data. Suites of such validation tests can be used to clearly identify the merits and limitations of existing models and developmental progress on new models.Here we describe NeuronUnit, a library that builds upon SciUnit and integrates with several existing neuroinformatics resources to support the validation of single-neuron models using data gathered by neurophysiologists and neuroanatomists. NeuronUnit integrates with existing technologies like Jupyter, Pandas, NeuroML and resources such as NeuroElectro, The Allen Institute, and The Human Brain Project in order to make neuron model validation as easy as possible for computational neuroscientists.

NeuroML-DB: Sharing and characterizing data-driven neuroscience models described in NeuroML

“…These neuron model characterizations are available as online tables (Fig 2A). Furthermore, to facilitate reuse of these properties, we implemented them as standardized tests within the SciUnit/NeuronUnit framework [70].…”

“…Properties and Transformations. All 38 electrophysiology properties from [68] were defined using reusable, Python-based tests within the NeuronUnit [70] validation framework (the tests are available in [90]). Additionally, the following four properties were used in the PCA and clustering analysis.…”

Bridging model and experiment in systems neuroscience with Cleo: the Closed-Loop, Electrophysiology, and Optophysiology simulation testbed

“…Using Cleo to compare the results of a simulation more directly with experimental data allows a more useful bridge between theory and experiment than the typical practice of using artificial inputs and ground-truth model outputs instead of inputs and measurements that more closely mimic an electrophysiology experiment. Subsequent comparison of simulation to experiment would allow the user to evaluate the model in the spirit of NeuronUnit [89], NetworkUnit [90, 91], and other such tools [92].…”

Model Integration in Computational Biology: The Role of Reproducibility, Credibility and Utility

“…For example, this establishes the domain under which their clinical use would be supported. When considering test implementation, some suggestions emphasize the need for a structured approach with unit-test style tests (Sarma et al, 2016;Gerkin et al, 2019) (Lieven et al, 2020) and continuous evaluation of such tests similar to continuous integration of software (Meyer, 2014;Krafczyk et al, 2019;Zhao et al, 2017). To enhance model credibility further, the model description should include validation tests against independent data, uncertainty assessments, and peer reviews (Refsgaard et al, 2005;Jakeman et al, 2006).…”