Tracking cell lineages in 3D by incremental deep learning

Abstract

Deep learning is emerging as a powerful approach for bioimage analysis, but its wider use is limited by the scarcity of annotated data for training. We present ELEPHANT, an interactive platform for cell tracking in 4D that seamlessly integrates annotation, deep learning, and proofreading. ELEPHANT's user interface supports cycles of incremental learning starting from sparse annotations, yielding accurate, user-validated cell lineages with a modest investment in time and effort.

Automated Reconstruction of Whole-Embryo Cell Lineages by Learning from Sparse Annotations

“…Because TGMM cannot process multi-channel input, for ZF we produced tracks for each of the two views separately, and reported the best result for each evaluation region. More recently, the tracking method included in the ELEPHANT framework has potential to be scalable to multi-terabyte datsets (Sugawara et al, 2021).…”

“…While handengineered features are sufficient for cell detection and tracking in some model organisms (Bao et al, 2006;Amat et al, 2014), learned dataset-specific features, given sufficient training data, improve performance for datasets with heterogeneous cell or nucleus phenotypes and varying imaging statistics over time and space. In particular, deep learning has been shown to improve cell detection (Kok et al, 2020;Hayashida et al, 2020), segmentation (Weigert et al, 2020;Cao et al, 2020;Stringer et al, 2021;Medeiros et al, 2021), and tracking (Sugawara et al, 2021;Ulman et al, 2017;Moen et al, 2019;Hayashida et al, 2020;Medeiros et al, 2021) on a variety of datasets. Additionally, it has been shown that tracking methods that take into account global spatiotemporal context perform better, especially for datasets with more movement between time frames (Ulman et al, 2017).…”

“…Of methods that fulfill these requirements, Tracking with Gaussian Mixture Models (TGMM) (Amat et al, 2014) has been shown to work well on model organisms with approximately ellipsoid nuclei. More recently, the ELEPHANT tracking platform employed deep learning for cell detection and per-frame linking in light-sheet datasets with diverse cell appearance and movement (Sugawara et al, 2021). ELEPHANT requires a manual pseudo-segmentation of nuclei by ellipsoid fitting, which takes less time to generate than a per-pixel manual segmentation, but more than point annotations.…”

“…Because we are learning features from the data, the method is not tied to a specific input type or format: we use fused and unfused lightsheet recordings with a single fluorescent nuclear channel, and could easily extend to multi-channel input. We use sparse point annotations to train a convolutional neural network to predict at each pixel a cell indicator value that peaks at the center of each nucleus (Höfener et al, 2018;Kok et al, 2020), and a movement vector that points to the center of the same cell nucleus in the previous time frame (Hayashida et al, 2020;Sugawara et al, 2021). From these predictions, we generate a candidate graph in two steps: first, we place nodes at the local maxima of the cell indicator values to represent possible cell center locations, with a score to encode the network's confidence.…”

“…We evaluate our method on three sparsely annotated datasets from different commonly used model organisms to study embryogenesis: mouse (McDole et al, 2018), Drosophila (Amat et al, 2014), and zebrafish (Wan et al, 2019b) (see Supplementary Note 1 for details about the datasets and annotations). We compare the performance of our method against TGMM, the previous state-of-the-art method on these datasets (Amat et al, 2014;McDole et al, 2018), and greedy tracking using a per-frame nearest neighbor linking algorithm similar to the ELEPHANT tracking method (Sugawara et al, 2021). We compute multiple metrics, including the fraction of perfectly constructed lineages over a range of time periods, and errors per ground truth edge, broken into the following error types: false negative edges (FN), identity switches(IS)-when two tracks switch off following the same cell-false positive divisions (FP-D), and false negative divisions (FN-D), as illustrated in Fig.…”