Collaborative infrastructure for test-driven scientific model validation

Omar, Cyrus; Aldrich, Jonathan; Gerkin, Richard C.

doi:10.1145/2591062.2591129

Cited by 30 publications

(43 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…T2: Scientific software developers benefit from using a wide range of testing practices from software engineering. Twelve studies made this claim [19,28,37,[39][40][41][42][43][44][45][46][47]. One method of addressing the problem in T1 is to use test-driven development to keep bugs such as these from remaining in their code in addition to doing a regular, automated build in order to test their code on a regular basis rather than waiting until project is completed [19,37,39,42].…”

Section: Testingmentioning

confidence: 99%

Claims about the use of software engineering practices in science: A systematic literature review

Heaton

Carver

2015

Information and Software Technology

View full text Add to dashboard Cite

a b s t r a c tContext: Scientists have become increasingly reliant on software in order to perform research that is too timeintensive, expensive, or dangerous to perform physically. Because the results produced by the software drive important decisions, the software must be correct and developed efficiently. Various software engineering practices have been shown to increase correctness and efficiency in the development of traditional software. It is unclear whether these observations will hold in a scientific context.Objective: This paper evaluates claims from software engineers and scientific software developers about 12 different software engineering practices and their use in developing scientific software. Method:We performed a systematic literature review examining claims about how scientists develop software. Of the 189 papers originally identified, 43 are included in the literature review. These 43 papers contain 33 different claims about 12 software engineering practices. Results:The majority of the claims indicated that software engineering practices are useful for scientific software development. Every claim was supported by evidence (i.e. personal experience, interview/survey, or case study) with slightly over half supported by multiple forms of evidence. For those claims supported by only one type of evidence, interviews/surveys were the most common. The claims that received the most support were: "The effectiveness of the testing practices currently used by scientific software developers is limited" and "Version control software is necessary for research groups with more than one developer." Additionally, many scientific software developers have unconsciously adopted an agile-like development methodology.Conclusion: Use of software engineering practices could increase the correctness of scientific software and the efficiency of its development. While there is still potential for increased use of these practices, scientific software developers have begun to embrace software engineering practices to improve their software. Additionally, software engineering practices still need to be tailored to better fit the needs of scientific software development.

show abstract

Section: Testingmentioning

confidence: 99%

Claims about the use of software engineering practices in science: A systematic literature review

Heaton

Carver

2015

Information and Software Technology

View full text Add to dashboard Cite

show abstract

“…Professional software developers face similar issues 3 . They must understand the scope of each component of a complex software project and verify that each component achieves desired input/output behavior.…”

Section: The Solution: Unit Testing For Modelsmentioning

confidence: 99%

NeuronUnit: A package for data-driven validation of neuron models using SciUnit

Gerkin

Birgiolas

Jarvis

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

Not peer-reviewed ABSTRACTValidating a quantitative scientific model requires comparing its predictions against many experimental observations, ideally from many labs, using transparent, robust, statistical comparisons. Unfortunately, in rapidly-growing fields like neuroscience, this is becoming increasingly untenable, even for the most conscientious scientists. Thus the merits and limitations of existing models, or whether a new model is an improvement on the state-of-the-art, is often unclear.Software engineers seeking to verify, validate and contribute to a complex software project rely on suites of simple executable tests, called "unit tests". Drawing inspiration from this practice, we previously developed SciUnit, an easy-to-use framework for developing data-driven "model validation tests" -executable functions, here written in Python. Each such test generates and statistically validates predictions from a model against one relevant feature of empirical data to produce a score indicating agreement between the model and the data. Suites of such validation tests can be used to clearly identify the merits and limitations of existing models and developmental progress on new models.Here we describe NeuronUnit, a library that builds upon SciUnit and integrates with several existing neuroinformatics resources to support the validation of single-neuron models using data gathered by neurophysiologists and neuroanatomists. NeuronUnit integrates with existing technologies like Jupyter, Pandas, NeuroML and resources such as NeuroElectro, The Allen Institute, and The Human Brain Project in order to make neuron model validation as easy as possible for computational neuroscientists.

show abstract

“…Furthermore, building a computational model is in the end a software development project of sort. Omar et al [39] have therefore proposed a framework for automated validation of scientific models, SciUnit, which is based on unit test- Figure 1: Schematic representation of the SciUnit framework. Models can be tested against experimental observations using specific tests.…”

Section: Introductionmentioning

confidence: 99%

“…the set of observable quantities that it can generate predictions about) of the model explicit and by allowing its validity (i.e. the extent to which its predictions agree with available experimental observations of those quantities) to be automatically tested [39].…”

Section: Introductionmentioning

confidence: 99%

Modules for Automated Validation and Comparison of Models of Neurophysiological and Neurocognitive Biomarkers of Psychiatric Disorders: ASSRUnit—A Case Study

Metzner

Mäki‐Marttunen

Zurowski

et al. 2018

Computational Psychiatry

View full text Add to dashboard Cite

The characterisation of biomarkers and endophenotypic measures has been a central goal of research in psychiatry over the last years. While most of this research has focused on the identification of biomarkers and endophenotypes, using various experimental approaches, it has been recognised that their instantiations, through computational models, have a great potential to help us understand and interpret these experimental results. However, the enormous increase in available neurophysiological and neurocognitive as well as computational data also poses new challenges. How can a researcher stay on top of the experimental literature? How can computational modelling data be efficiently compared to experimental data? How can computational modelling most effectively inform experimentalists? Recently, a general scientific framework for the generation of executable tests that automatically compare model results to experimental observations, SciUnit, has been proposed. Here we exploit this framework for research in psychiatry to address the challenges mentioned above. We extend the SciUnit framework by adding an experimental database, which contains a comprehensive collection of relevant experimental observations, and a prediction database, which contains a collection of predictions generated by computational models. Together with appropriately designed SciUnit tests and methods to mine and visualise the databases, model data and test results, this extended framework has the potential to greatly facilitate the use of computational models in psychiatry. As an initial example we present ASSRUnit, a module for auditory steady-state response deficits in psychiatric disorders.

show abstract

Collaborative infrastructure for test-driven scientific model validation

Cited by 30 publications

References 6 publications

Claims about the use of software engineering practices in science: A systematic literature review

Claims about the use of software engineering practices in science: A systematic literature review

NeuronUnit: A package for data-driven validation of neuron models using SciUnit

Modules for Automated Validation and Comparison of Models of Neurophysiological and Neurocognitive Biomarkers of Psychiatric Disorders: ASSRUnit—A Case Study

Contact Info

Product

Resources

About