Dense cellular segmentation for EM using 2D-3D neural network ensembles

Abstract

Modern biological electron microscopy produces nanoscale images from biological samples of unprecedented volume, and researchers now face the problem of making use of the data. Image segmentation has played a fundamental role in EM image analysis for decades, but challenges from biological EM have spurred interest and rapid advances in computer vision for automating the segmentation process. In this paper, we demonstrate dense cellular segmentation as a method for generating rich 3D models of tissues and their constituent cells and organelles from scanning electron microscopy images. We describe how to use ensembles of 2D-3D neural networks to compute dense cellular segmentations of cells and organelles inside two human platelet tissue samples. We conclude by discussing ongoing challenges for realizing practical dense cellular segmentation algorithms. The data and code used in this paper, as well as example notebooks, are available at leapmanlab. github.io/dense-cell.Biomedical researchers use electron microscopy (EM) to image cells, organelles, and their constituents at the nanoscale. The serial block-face scanning electron microscope (SBF-SEM) 1 uses automated serial sectioning techniques to rapidly produce gigavoxel image volumes and beyond. This rapid growth in throughput challenges traditional image analytic workflows for EM, which rely on trained humans to identify salient image features. Highthroughput EM offers to revolutionize systems biology by providing nanoscale structural detail across macroscopic tissue regions, but analyses of such datasets in their entirety will be infeasibly expensive and time-consuming until analytic bottlenecks are automated. A fundamental component of common EM image anaylsis workflows is segmentation, which groups image voxels together into labeled regions that correspond to image content. For semantic segmentation, each voxel is assigned an object classification label, such as cell or mitochondrion. In this paper we introduce a dense cellular segmentation task, illustrated in Figure 1, which seeks to segment the entirety of an image volume with multiple, detailed class labels. Dense semantic segmentation is vital for systems biologists seeking to create semantically-rich 3D models of cells and subcellular structure interconnected within tissue environments. Manually performing dense segmentation tasks for EM volumes is tedious and infeasible at scale for new highthroughput microscopes. However, automating dense segmentation for EM is challenging due to the image complexity of biological structures at the nanoscale. An image with little noise and high contrast between features may be accurately segmented with simple methods such as thresholding 2 , while accurate segmentation of complicated images with multiscale features, noise, and textural content remains an open problem for many tasks of interest to biomedicine. Current state-of-the-art computational solutions use convolutional neural network architectures to solve problems on a per-task basis. The field of connectomics h...

CEM500K – A large-scale heterogeneous unlabeled cellular electron microscopy image dataset for deep learning

“…Before we were able to evaluate the MoCoV2 algorithm on CEM500K, it was necessary to define a set of downstream tasks to quantify and compare performance. We chose six publicly available benchmark datasets: CREMI Synaptic Clefts [42], Guay [15], Kasthuri++ and Lucchi++ [17], Perez [18] and UroCell [16]. The benchmarks included a total of eight organelles or subcellular structures for segmentation (mitochondria, lysosomes, nuclei, nucleoli, canalicular channels, alpha granules, dense granules, dense granule cores, and synaptic clefts).…”

“…Many of the limitations of supervised DL segmentation models for cellular EM data result from a lack of large and, importantly, diverse training datasets [12][13] [14]. Although several annotated image datasets for cell and organelle segmentation are publicly available, these often exclusively consist of images from a single experiment or tissue type, and a single imaging approach [15] [16] [17] [18] [19]. The homogeneity of such datasets often means that they are ineffective for training DL models to accurately segment images from unseen experiments.…”

“…Instead, when confronted with new data, the norm is to extract and annotate small regions-ofinterest (ROIs) from the EM image, train a model on the ROIs, and then apply the model to infer segmentations for the remaining unlabeled data [15] [16] [17] [18] [19] [20] [21]. Often, not only are these models dataset-specialized, reducing their utility, they often fail to generalize even to parts of the same dataset that are spatially distant from the training ROIs [16] [22].…”

CEM500K, a large-scale heterogeneous unlabeled cellular electron microscopy image dataset for deep learning

“…Many of the limitations of supervised DL segmentation models for cellular EM data result from a lack of large and, importantly, diverse training datasets ( Goodfellow, 2016 ; Pereira et al, 2009 ; Sun et al, 2017 ). Although several annotated image datasets for cell and organelle segmentation are publicly available, these often exclusively consist of images from a single experiment or tissue type, and a single imaging approach ( Guay et al, 2020 ; Žerovnik Mekuč et al, 2020 ; Casser et al, 2018 ; Perez et al, 2014 ; Berning et al, 2015 ). The homogeneity of such datasets often means that they are ineffective for training DL models to accurately segment images from unseen experiments.…”

“…The homogeneity of such datasets often means that they are ineffective for training DL models to accurately segment images from unseen experiments. Instead, when confronted with new data, the norm is to extract and annotate small regions-of-interest (ROIs) from the EM image, train a model on the ROIs, and then apply the model to infer segmentations for the remaining unlabeled data ( Guay et al, 2020 ; Žerovnik Mekuč et al, 2020 ; Casser et al, 2018 ; Perez et al, 2014 ; Berning et al, 2015 ; Januszewski et al, 2018 ; Funke et al, 2019 ). Often, not only are these models dataset-specialized, reducing their utility, they often fail to generalize even to parts of the same dataset that are spatially distant from the training ROIs ( Žerovnik Mekuč et al, 2020 ; Buhmann, 2019 ).…”