Petascale pipeline for precise alignment of images from serial section electron microscopy

Abstract

The reconstruction of neural circuits from serial section electron microscopy (ssEM) images is being accelerated by automatic image segmentation methods. Segmentation accuracy is often limited by the preceding step of aligning 2D section images to create a 3D image stack. Precise and robust alignment in the presence of image artifacts is challenging, especially as datasets are attaining the petascale. We present a computational pipeline for aligning ssEM images with several key elements. Self-supervised convolutional nets are trained via metric learning to encode and align image pairs, and they are used to initialize iterative fine-tuning of alignment. A procedure called vector voting increases robustness to image artifacts or missing image data. For speedup the series is divided into blocks that are distributed to computational workers for alignment. The blocks are aligned to each other by composing transformations with decay, which achieves a global alignment without resorting to a time-consuming global optimization. We apply our pipeline to a whole fly brain dataset, and show improved accuracy relative to prior state of the art. We also demonstrate that our pipeline scales to a cubic millimeter of mouse visual cortex. Our pipeline is publicly available through two open source Python packages.

Fast imaging of millimeter-scale areas with beam deflection transmission electron microscopy

“…4d). This is not surprising, given that downsampled images at 8 or 16 nm per pixel have been shown to be adequate for high-quality alignment and segmentation 46,47 . These data demonstrate that sequential sections imaged with the bdTEMs can be assembled into a 3D volume to reconstruct neural morphology and connectivity.…”

“…Adding more subtiles should further increase overall imaging speed but return would be diminished given the proportion of stage movement overhead is already down to 4% of total time, with trade-offs of lower SNRs for the tiles located at the peripheries. (5) Given that downsampled data (for instance, at a pixel size of 8 or 16 nm) have proved to be adequate for high-quality alignment and segmentation 46,47 , one potential approach to increase imaging speed per unit area of tissue is to either increase pixel size (i.e. microscope magnification) or further decrease exposure time, with the caveat that the loss in information content be carefully evaluated against groundtruth.…”

“…In fact, fine alignments of volume cutouts show no difference between images acquired with 120 ms and 40 ms exposure time (Supplementary Video 1-2). Given that alignment quality is a major limiting factor influencing segmentation accuracy 46 , imaging at 40 ms exposure time should be sufficient. In ongoing large-scale imaging efforts, we are routinely using exposure time of around 40 ms.…”

“…Sectioning-related artifacts (folds, cracks, knife marks, etc.) are very common, though automated reconstruction methods that are robust to some defects have been developed 46,47 . Wide-area milling of imaged tissue was recently developed to circumvent ultrathin sectioning 15,16 and, if integrated with a MultiSEM, can potentially scale up connectomic volumes beyond petascale.…”

NeuVue: A Framework and Workflows for High-Throughput Electron Microscopy Connectomics Proofreading

“…As high-resolution electron microscopy (EM) data acquisition capabilities continue to advance, it has become common for research labs to image increasingly large (e.g., > 100 teravoxels) data volumes (Shapson-Coe et al 2021;MICrONS Consortium et al 2021;Turner et al 2022;Dorkenwald et al 2019). Following image volume collection, automated methods are applied to segment and identify objects of interest, such as neurons, nuclei, and synapses (Januszewski et al 2018;Popovych et al 2022;Yin et al 2020). The performance of these algorithms has dramatically improved in the past few years; however, current methods are still imperfect and require significant human intervention (Scheffer et al 2020;Dorkenwald et al 2022).…”