Whole-cell segmentation of tissue images with human-level performance using large-scale data annotation and deep learning

Abstract

A major challenge in the analysis of tissue imaging data is cell segmentation, the task of identifying the precise boundary of every cell in an image. To address this problem we constructed TissueNet, a dataset for training segmentation models that contains more than 1 million manually labeled cells, an order of magnitude more than all previously published segmentation training datasets. We used TissueNet to train Mesmer, a deep learning-enabled segmentation algorithm. We demonstrated that Mesmer is more accurate than previous methods, generalizes to the full diversity of tissue types and imaging platforms in TissueNet, and achieves human-level performance. Mesmer enabled the automated extraction of key cellular features, such as subcellular localization of protein signal, which was challenging with previous approaches. We then adapted Mesmer to harness cell lineage information in highly multiplexed datasets and used this enhanced version to quantify cell morphology changes during human gestation. All code, data, and models are released as a community resource.

Transition to invasive breast cancer is associated with progressive changes in the structure and composition of tumor stroma

“…Cell segmentation was performed on pre-processed images using deep learning based software previously developed in our lab, Mesmer ( Greenwald et al, 2021b ), publicly available at https://www.deepcell.org/predict . The input to Mesmer is a two-channel image containing a nuclear marker in one channel and membrane or cytoplasmic markers in the other to accurately delineate single cell nuclei.…”

Multicellular modules as clinical diagnostic and therapeutic targets

“…We expect that computational advances will also facilitate the identification and dissection of complex multicellular modules that can serve as the basis for new diagnostics and therapeutics. Machine learning is used to segment out individual cells in images for single cell quantification [26] and to quantify lower-plex immunohistochemistry data [27]. Machine learning approaches could be extended to enable segmentation of complex, multicellular structures, such as the glomeruli of the kidney, and to automate cell type annotation.…”

Deep Learning and Its Applications in Computational Pathology

“…With the introduction of GAN, using conditional-GAN or cycle-GAN models and in combination with CNN models for segmentation problems is also shown to be viable, with less stringent training data requirements [46,53,77]. Unlike most classification models, the segmentation models can be more adaptive to different types of tissues due to the similarities of the stained features and textures of the histopathology slides [78]. Additionally, the evaluation metrics of these classification models can be drastically different from those of the classification models.…”