Community-based benchmarking improves spike rate inference from two-photon calcium imaging data

Abstract

In recent years, two-photon calcium imaging has become a standard tool to probe the function of neural circuits and to study computations in neuronal populations. However, the acquired signal is only an indirect measurement of neural activity due to the comparatively slow dynamics of fluorescent calcium indicators. Different algorithms for estimating spike rates from noisy calcium measurements have been proposed in the past, but it is an open question how far performance can be improved. Here, we report the results of the spikefinder challenge, launched to catalyze the development of new spike rate inference algorithms through crowd-sourcing. We present ten of the submitted algorithms which show improved performance compared to previously evaluated methods. Interestingly, the top-performing algorithms are based on a wide range of principles from deep neural networks to generative models, yet provide highly correlated estimates of the neural activity. The competition shows that benchmark challenges can drive algorithmic developments in neuroscience.

A zero-inflated gamma model for post-deconvolved calcium imaging traces

“…Next, in Figure 3, we examine data shared through the SpikeFinder challenge (Berens et al, 2018), including traces recorded using four calcium indicators (GCamp6s, jRCAMP1a, OGB-1, jRGECO1a). Again, we find that the ZIG model provided a good fit across a wide range of data.…”

“…At the same time, calcium imaging presents some important analysis challenges: calcium signals represent a slow, nonlinear encoding of the underlying spike train signals of interest, and therefore it is necessary to denoise and temporally deconvolve temporal traces extracted from calcium video data into estimates of neural activity. These issues have received extensive attention in the literature (Vogelstein et al, 2009;Vogelstein et al, 2010;Pnevmatikakis et al, 2016;Deneux et al, 2016;Theis et al, 2016;Friedrich et al, 2017;Speiser et al, 2017;Aitchison et al, 2017;Berens et al, 2018;Pachitariu et al, 2018;Greenberg et al, 2018). Some of these deconvolution approaches estimate spiking probabilities directly (Vogelstein et al, 2009;Pnevmatikakis et al, 2016;Deneux et al, 2016;Speiser et al, 2017;Aitchison et al, 2017;Greenberg et al, 2018), but many approaches instead estimate the influx of calcium in each time bin, rather than a spiking probability (Vogelstein et al, 2010;Pnevmatikakis et al, 2016;Friedrich et al, 2017;Berens et al, 2018;Pachitariu et al, 2018;Stringer and Pachitariu, 2019); these non-probabilistic approaches tend to be faster and are therefore popular in practice.…”

SpikeForest: reproducible web-facing ground-truth validation of automated neural spike sorters

“…To our knowledge, these unmaintained projects are either small-scale snapshots or are only partially realized. Yet, in the related area of calcium imaging, leaderboard-style comparison efforts have been more useful for establishing community benchmarks (Freeman, 2015(Freeman, -2018Berens et al, 2018).…”

“…We constrast this to competition-style efforts (Franke et al, 2012;Freeman, 2015Freeman, -2018Berens et al, 2018) which allow contributions of (potentially non-reproducible) sorting results, and which report accuracy on held-out data whose ground truth are necessarily private, and thus cannot be interrogated by the community. Our work has three main objectives.…”

DeepCINAC: a deep-learning-based Python toolbox for inferring calcium imaging neuronal activity based on movie visualization

“…To meet this challenge, we have developed a method based on deeplearning. Even though, several deep-learning based methods to infer neuronal activity from fluorescence traces already exist 12 , none of them proposes a method directly using two-photon recordings.…”