Modules for Automated Validation and Comparison of Models of Neurophysiological and Neurocognitive Biomarkers of Psychiatric Disorders: ASSRUnit—A Case Study

Abstract

The characterisation of biomarkers and endophenotypic measures has been a central goal of research in psychiatry over the last years. While most of this research has focused on the identification of biomarkers and endophenotypes, using various experimental approaches, it has been recognised that their instantiations, through computational models, have a great potential to help us understand and interpret these experimental results. However, the enormous increase in available neurophysiological and neurocognitive as well as computational data also poses new challenges. How can a researcher stay on top of the experimental literature? How can computational modelling data be efficiently compared to experimental data? How can computational modelling most effectively inform experimentalists? Recently, a general scientific framework for the generation of executable tests that automatically compare model results to experimental observations, SciUnit, has been proposed. Here we exploit this framework for research in psychiatry to address the challenges mentioned above. We extend the SciUnit framework by adding an experimental database, which contains a comprehensive collection of relevant experimental observations, and a prediction database, which contains a collection of predictions generated by computational models. Together with appropriately designed SciUnit tests and methods to mine and visualise the databases, model data and test results, this extended framework has the potential to greatly facilitate the use of computational models in psychiatry. As an initial example we present ASSRUnit, a module for auditory steady-state response deficits in psychiatric disorders.

NeuronUnit: A package for data-driven validation of neuron models using SciUnit

“…Recognizing this, and inspired by unit testing practices, we previously developed a discipline-agnostic framework for developing model validation test suites called SciUnit 3 (available from http:// sciunit.scidash.org). SciUnit is now used in over a dozen scientific projects 5,6,[13][14][15][16][17][18][19][20][21][22][23][24][25][26][27][28][29] including The Human Brain Project 30, 31 .…”

“…After some agreement is reached about protocol design (how long of a square pulse, whether to do a series of fixed current steps or a binary search, what resolution to use, etc.) that protocol can be implemented (lines [15][16][17][18][19][20][21][22][23][24][25][26][27][28][29][30][31][32][33][34], and goodness-of-fit can be measured by calculating a z-score (among other possibilities) given by the difference between the predicted rheobase and the mean observed rheobase (e.g. across cells) divided by the standard deviation of the observed rheobase.…”

“…a membrane potential time series) and returns a Neo SpikeTrain object ( Fig. 4, lines [20][21][22][23][24][25][26]. This reduces the problem of generating the spike train to one of obtaining a membrane potential time series, which might plausibly vary across models.…”

“…2, line 57) checks that these capabilities are implemented by the model, uses them to extract a prediction via get prediction (Fig. 2, lines [19][20][21][22][23][24][25][26][27][28][29][30][31][32][33][34], to compute a score indicating model/experiment agreement (not shown) and return a score. In the example from Fig.…”

“…Meanwhile, corresponding test suite repositories on GitHub, containing NeuronUnit-based tests, do not even reach this stage (with few exceptions 21 ). However, many neuroscience projects are realizing this vision using SciUnit directly 5,6,[15][16][17][18][19][20][24][25][26][27][28][29] . We plan to continue creating such repositories; more importantly we invite interested parties to create their own, either to guide in-house model development or to assess larger pools of established models.…”