Segmentation-Enhanced CycleGAN

Abstract

Algorithmic reconstruction of neurons from volume electron microscopy data traditionally requires training machine learning models on dataset-specific ground truth annotations that are expensive and tedious to acquire. We enhanced the training procedure of an unsupervised image-to-image translation method with additional components derived from an automated neuron segmentation approach. We show that this method, Segmentation-Enhanced CycleGAN (SECGAN), enables near perfect reconstruction accuracy on a benchmark connectomics segmentation dataset despite operating in a "zero-shot" setting in which the segmentation model was trained using only volumetric labels from a different dataset and imaging method. By reducing or eliminating the need for novel ground truth annotations, SECGANs alleviate one of the main practical burdens involved in pursuing automated reconstruction of volume electron microscopy data. Neural network architecturesWe used a 3d generalization of ResNet-18 24 as the architecture for all three discriminator networks, and a stack of 8 residual convolutional modules as the architecture of the generator networks. We also tested a U-net-style generator, as well as a simple convolution-pooling network as the discriminator, and found these alternatives resulted in worse segmentation quality.

A connectome and analysis of the adult Drosophila central brain

“…paintings), without being given any explicit pairings between elements of both sets. Here we extended this method to 3D volumes, and used model architectures and training hyperparameters as previously described (Januszewski and Jain, 2019), but without utilizing the flood-filling module.…”

Beyond the feedforward sweep: feedback computations in the visual cortex

“…In a similar vein, Linsley et al . have shown that the network architectures that exhibit “superhuman” accuracy for the segmentation of neural tissue from serial electron microscopy images when trained and tested on different subsets of the same volume do exhibit a large drop in accuracy when trained and tested on different volumes 81 (for practical applications, the issue can be alleviated using machine learning methods for “realigning” datasets 82 ). By comparison, they found that RNNs endowed with horizontal and top‐down connections can generalize much better and use fewer training examples 66,77 …”

Exploiting prior knowledge about biological macromolecules in cryo-EM structure determination

“…Similarly, cycle-consistent adversarial networks (CycleGAN) [40] or style transfer GANs [41] may be used to relate the two data domains in cases where pairs are lacking, e.g. for low resolution reconstructions without a matching ground truth or for generalizing the data set to new experimental conditions with few existing examples [42]. Thirdly, the approach described here may be adapted to use optimisation algorithms other than expectation maximisation.…”