Neural Reconstruction Integrity: A Metric for Assessing the Connectivity Accuracy of Reconstructed Neural Networks

Abstract

Neuroscientists are actively pursuing high-precision maps, or graphs consisting of networks of neurons and connecting synapses in mammalian and non-mammalian brains. Such graphs, when coupled with physiological and behavioral data, are likely to facilitate greater understanding of how circuits in these networks give rise to complex information processing capabilities. Given that the automated or semi-automated methods required to achieve the acquisition of these graphs are still evolving, we developed a metric for measuring the performance of such methods by comparing their output with those generated by human annotators (“ground truth” data). Whereas classic metrics for comparing annotated neural tissue reconstructions generally do so at the voxel level, the metric proposed here measures the “integrity” of neurons based on the degree to which a collection of synaptic terminals belonging to a single neuron of the reconstruction can be matched to those of a single neuron in the ground truth data. The metric is largely insensitive to small errors in segmentation and more directly measures accuracy of the generated brain graph. It is our hope that use of the metric will facilitate the broader community's efforts to improve upon existing methods for acquiring brain graphs. Herein we describe the metric in detail, provide demonstrative examples of the intuitive scores it generates, and apply it to a synthesized neural network with simulated reconstruction errors. Demonstration code is available.

Leveraging Tools from Autonomous Navigation for Rapid, Robust Neuron Connectivity

“…Graph integrity was measured based on neural reconstruction integrity (NRI) [9], a metric that measures graph similarity according to precision and recall of intracellular paths. Incorrect edges (additions or deletions) are penalized and used to create a precision and recall score reflective of graph quality.…”

DotMotif: An open-source tool for connectome subgraph isomorphism search and graph queries

“…Neuroscience questions may be reformulated as analyses on a graph, and the neuroscientist may add graph theory to the toolbox of strategies with which to understand the brain [6,9]. Such questions include searching for specific subgraph structures, investigating the connectivity of specific neuron cell types or categories, proofreading connectomes for accuracy, and generating summary statistics on the graph as a whole [10,11,2,12,13,14,8].…”

Analyzing Image Segmentation for Connectomics

“…We will show later that evaluating segmentation around synapses more directly results in less optimistic scoring compared to traditional metrics like (Meilă, 2003 ). Recent work in Reilly et al ( 2017 ) also introduced a metric that more appropriately weighs the impact of synapses on segmentation, though it does not explicitly consider connectivity correctness between neurons.…”

“…One potential solution is to define S and G in Equation 1 over a set of exemplar points representing synapses, instead of all segmentation voxels as done in Plaza ( 2016 ) and Plaza and Berg ( 2016 ). A similar strategy of measuring groupings of synapses was introduced in Reilly et al ( 2017 ), which additionally breaks down results per neuron making the results more interpretable. While these metrics better emphasize correctness near synapses, it is not obvious how to interpret error impact to connectivity pathways.…”

“…Presumably, this becomes a bigger concern when evaluating larger datasets. Notably, the authors in Reilly et al ( 2017 ) recognized this challenge and describe a metric that allows intuitive insights at the neuron level. In Nunez-Iglesias et al ( 2013 ), the authors decompose the VI calculation to provide scores per 3D segment.…”