ImmuNet: A Segmentation-Free Machine Learning Pipeline for Immune Landscape Phenotyping in Tumors by Muliplex Imaging

Abstract

Tissue specimens taken from primary tumors or metastases contain important information for diagnosis and treatment of cancer patients. Multispectral imaging allows in situ visualization of heterogeneous cell subsets, such as lymphocytes, in tissue samples. Many image processing pipelines first segment cell boundaries and then measure marker expression to assign cell phenotypes. In dense tissue environments such as solid tumors, segmentation-based phenotyping can be inaccurate due to segmentation errors or overlapping cell boundaries. Here we introduce a machine learning pipeline design called ImmuNet that directly identifies the positions and phenotypes of immune cells without determining their exact boundaries. ImmuNet is easy to train: human annotators only need to click on immune cells and rank their expression of each marker; full annotation of tissue regions is not necessary. We demonstrate that ImmuNet is a suitable approach for immune cell detection and phenotyping in multiplex immunohistochemistry: it compares favourably to segmentation-based methods, especially in dense tissues, and we externally validate ImmuNet results by comparing them to flow cytometric measurements from the same tissue. In summary, ImmuNet performs well on diverse tissue specimens, takes relatively little effort to train and implement, and is a simpler alternative to segmentation-based approaches when only cell positions and phenotypes, but not their shapes are required for downstream analyses. We hope that ImmuNet will help cancer researchers to analyze multichannel tissue images more easily and accurately.

The immune microenvironment landscape shows treatment-specific differences in rectal cancer patients

“…Whole slide scans were acquired using x4 magnification and subsequently scanned at x20 magnification for the multispectral regions of interest ( Figure 1B ). Images were first processed using the inForm software (V.2.4.8, Akoya Biosciences, MA, USA, RRID: SCR_019155) for cell and tissue segmentation and then processed through an in-house AI pipeline to phenotype immune cells, which were thereafter analyzed in FlowJo™ (Ashland, OR, USA, RRID: SCR_008520) as previously described ( 35 , 36 ).…”

Tumor microenvironment shows an immunological abscopal effect in patients with NSCLC treated with pembrolizumab-radiotherapy combination

“…Details are described in the ImmuNet preprint. 21 The version of ImmuNet used to detect lymphocytes and the input images/ROIs will be deposited on Zenodo on acceptance of this manuscript.…”

Homologous recombination repair deficient prostate cancer represents an immunologically distinct subtype

“…After tissue segmentation, images and tissue segmentation data were exported from inForm for cell identification and phenotyping by an in-house developed neural network (ImmuNet). 31 Shortly, the neural network identifies TILs based on the expression of the seven IHC markers and predicts for each identified cell which of the markers is expressed (supplementary figure 1E-F). The data generated by the neural network were exported in Flow Cytometry Standard (FCS) files.…”