Automated neuron tracking inside moving and deforming animals using deep learning and targeted augmentation

Abstract

Advances in functional brain imaging now allow sustained rapid 3D visualization of large numbers of neurons inside behaving animals. To decode circuit activity, imaged neurons must be individually segmented and tracked. This is particularly challenging when the brain itself moves and deforms inside a flexible body. The field has lacked general methods for solving this problem effectively. To address this need, we developed a method based on a convolutional neural network (CNN) with specific enhancements which we apply to freely moving Caenorhabditis elegans. For a traditional CNN to track neurons across images of a brain with different postures, the CNN must be trained with ground truth (GT) annotations of similar postures. When these postures are diverse, an adequate number of GT annotations can be prohibitively large to generate manually. We introduce 'targeted augmentation', a method to automatically synthesize reliable annotations from a few manual annotations. Our method effectively learns the internal deformations of the brain. The learned deformations are used to synthesize annotations for new postures by deforming the manual annotations of similar postures in GT images. The technique is germane to 3D images, which are generally more difficult to analyze than 2D images. The synthetic annotations, which are added to diversify training datasets, drastically reduce manual annotation and proofreading. Our method is effective both when neurons are represented as individual points or as 3D volumes. We provide a GUI that incorporates targeted augmentation in an end-to-end pipeline, from manual GT annotation of a few images to final proofreading of all images. We apply the method to simultaneously measure activity in the second-layer interneurons in C. elegans: RIA, RIB, and RIM, including the RIA neurite. We find that these neurons show rich behaviors, including switching entrainment on and off dynamically when the animal is exposed to periodic odor pulses.

Versatile Multiple Object Tracking in Sparse 2D/3D Videos Via Diffeomorphic Image Registration

“…Two step (detect and link) approaches often suffer from the lack of reliable detection algorithms and require relatively low frame-to-frame motion in order to accurately link the detected neurons (6, 12, 16). Similarly, deep learning approaches are limited by insufficient training data, often failing to generalize across different animals, even those within the same strain (11, 26, 46). While these approaches have provided important insight and progress, there remains substantial need for improvement in accuracy and efficiency when tracking many neurons in freely behaving worms.…”

“…A k-medoids clustering algorithm is applied to these pairwise distances to identify a small number of median frames to best serve as reference frames for all other frames in the corresponding cluster (Fig. 2) (4, 26).…”

“…In comparison, we test the flow field generation method used in taCNN (26). In this algorithm, neurons are tracked by analyzing images through two different convolutional neural networks.…”

A Graph Matching Approach to Tracking Neurons in Freely-MovingC. elegans

“…It is more difficult to achieve better results with the latter metric, as it is the small pieces of neurites that are hard to match. Another noteworthy aspect is that it takes 8-12 hours to train the model of Park et al (2022) on a single video with a GPU, whereas our method is faster, parallelizable, and runs on the CPU; see Appendix D for details on the runtimes. Future directions.…”

Neuron tracking in C. elegans through automated anchor neuron localization and segmentation

“…The MLP consists of linear layers with activation functions, performing transformations like following (5) on the absolution difference between each pair of node embedding Emb i and Emb j…”