CosMIC: A Consistent Metric for Spike Inference from Calcium Imaging

Abstract

In recent years, the development of algorithms to detect neuronal spiking activity from two-photon calcium imaging data has received much attention, yet few researchers have examined the metrics used to assess the similarity of detected spike trains with the ground truth. We highlight the limitations of the two most commonly used metrics, the spike train correlation and success rate, and propose an alternative, which we refer to as CosMIC. Rather than operating on the true and estimated spike trains directly, the proposed metric assesses the similarity of the pulse trains obtained from convolution of the spike trains with a smoothing pulse. The pulse width, which is derived from the statistics of the imaging data, reflects the temporal tolerance of the metric. The final metric score is the size of the commonalities of the pulse trains as a fraction of their average size. Viewed through the lens of set theory, CosMIC resembles a continuous Sørensen-Dice coefficient-an index commonly used to assess the similarity of discrete, presence/absence data. We demonstrate the ability of the proposed metric to discriminate the precision and recall of spike train estimates. Unlike the spike train correlation, which appears to reward overestimation, the proposed metric score is maximized when the correct number of spikes have been detected. Furthermore, we show that CosMIC is more sensitive to the temporal precision of estimates than the success rate.

Improved hyperacuity estimation of spike timing from calcium imaging

“…The inverse of spike distance (when the weighted cost of shifting spikes is proportional to the time window of spike detection) measured the temporal precision of detected spikes. The CosMIC metric assessed the similarity of the estimations with the ground truth spike trains by convoluting the two spike trains with a smooth kernel 40 (see "Methods"). We found that CosMIC exhibited performance profiles quite similar to those for the F1 score, with a strong correlation (r = 0.96, see Fig.…”

“…We computed the CosMIC metric to determine spike time precision 40 . The ground truth and estimated membership functions – y(t) and y est (t) , respectively – were computed by convoluting the ground truth and estimated spike train – s(t) and s est (t) , respectively – with the triangle kernel p(t) : where e is the width of the triangle kernel p(t) , and is equal to the time window of the ROC analysis.…”

Community-based benchmarking improves spike rate inference from two-photon calcium imaging data

“…Metrics for spike rate inference are a matter of current debate in the literature—see for example refs. [ 5 , 25 ] for recent proposals.…”

A database and deep learning toolbox for noise-optimized, generalized spike inference from calcium imaging

“…F1-scores combine false positives and negatives 12 but are difficult to compare across datasets when the baseline spike rates vary (which is the case for our database). Other metrics try to combine the strengths of the correlation measure with a sensitivity to the correct number of spikes 70 but are less intuitive.…”