Analyzing Image Segmentation for Connectomics

Abstract

Automatic image segmentation is critical to scale up electron microscope (EM) connectome reconstruction. To this end, segmentation competitions, such as CREMI and SNEMI, exist to help researchers evaluate segmentation algorithms with the goal of improving them. Because generating ground truth is time-consuming, these competitions often fail to capture the challenges in segmenting larger datasets required in connectomics. More generally, the common metrics for EM image segmentation do not emphasize impact on downstream analysis and are often not very useful for isolating problem areas in the segmentation. For example, they do not capture connectivity information and often over-rate the quality of a segmentation as we demonstrate later. To address these issues, we introduce a novel strategy to enable evaluation of segmentation at large scales both in a supervised setting, where ground truth is available, or an unsupervised setting. To achieve this, we first introduce new metrics more closely aligned with the use of segmentation in downstream analysis and reconstruction. In particular, these include synapse connectivity and completeness metrics that provide both meaningful and intuitive interpretations of segmentation quality as it relates to the preservation of neuron connectivity. Also, we propose measures of segmentation correctness and completeness with respect to the percentage of “orphan” fragments and the concentrations of self-loops formed by segmentation failures, which are helpful in analysis and can be computed without ground truth. The introduction of new metrics intended to be used for practical applications involving large datasets necessitates a scalable software ecosystem, which is a critical contribution of this paper. To this end, we introduce a scalable, flexible software framework that enables integration of several different metrics and provides mechanisms to evaluate and debug differences between segmentations. We also introduce visualization software to help users to consume the various metrics collected. We evaluate our framework on two relatively large public groundtruth datasets providing novel insights on example segmentations.

Anatomy and activity patterns in a multifunctional motor neuron and its surrounding circuits

“…Even so, the completeness of our functional dataset in combination with our anatomical data makes for an attractive basis for simulation studies to arrive at a computational understanding of multifunctional neuronal circuits ( Real et al, 2017 ). Our data may also serve as a large-scale ground truth for EM segmentation algorithms ( Plaza and Funke, 2018 ). This study not only represents an important step in a combined approach to studying multifunctional circuits at the synaptic level, but also lays the groundwork for a comprehensive neuronal mapping of a whole ganglion that semi-autonomously processes local sensory information and controls segmental movement.…”

Automatic Detection of Synaptic Partners in a Whole-BrainDrosophilaEM Dataset

“…Second, the quality of a neuron segmentation should be evaluated not just based on overall topological correctness, but also on accuracy close to boundaries, in particular in the proximity of synaptic clefts. Neither of the two most commonly used metrics to evaluate neuron segmentations (expected run length and variation of information) are sensitive to small errors close to synaptic terminals (Plaza and Funke, 2018).…”

CEM500K, a large-scale heterogeneous unlabeled cellular electron microscopy image dataset for deep learning

“…While supervised deep learning (DL) models are effective at the segmentation of objects in natural images (e.g. of people, cars, furniture, and landscapes) ( Wang et al, 2020 ; Tao et al, 2020 ; Carion et al, 2020 ; He et al, 2020 ), they require significant human oversight and correction when applied to the organelles and cellular structures captured by EM ( Lichtman et al, 2014 ; Plaza and Funke, 2018 ).…”