Deep Learning-Inferred Multiplex ImmunoFluorescence for IHC Image Quantification

Abstract

Reporting biomarkers assessed by routine immunohistochemical (IHC) staining of tissue is broadly used in diagnostic pathology laboratories for patient care. To date, clinical reporting is predominantly qualitative or semi-quantitative. By creating a multitask deep learning framework referred to as DeepLIIF, we are presenting a single step solution to nuclear segmentation and quantitative single-cell IHC scoring. Leveraging a unique de novo dataset of co-registered IHC and multiplex immunoflourescence (mpIF) data generated from the same tissue section, we simultaneously segment and translate low-cost and prevalent IHC slides to more expensive-yet-informative mpIF images. Moreover, a nuclear-pore marker, LAP2beta, is co-registered to improve cell segmentation and protein expression quantification on IHC slides. By formulating the IHC quantification as cell instance segmentation/classification rather than cell detection problem, we show that our model trained on clean IHC Ki67 data can generalize to more noisy and artifact-ridden images as well as other nuclear and non-nuclear markers such as CD3, CD8, BCL2, BCL6, MYC, MUM1, CD10 and TP53. We thoroughly evaluate our method on publicly available benchmark datasets as well as against pathologists' semi-quantitative scoring. The code, trained models, and the resultant embeddings for all the datasets used in this paper will be released at https://github.com/nadeemlab/DeepLIIF.

Multicellular modules as clinical diagnostic and therapeutic targets

“…We expect that computational advances will also facilitate the identification and dissection of complex multicellular modules that can serve as the basis for new diagnostics and therapeutics. Machine learning is used to segment out individual cells in images for single cell quantification [26] and to quantify lower-plex immunohistochemistry data [27]. Machine learning approaches could be extended to enable segmentation of complex, multicellular structures, such as the glomeruli of the kidney, and to automate cell type annotation.…”

Quantification of the Immune Content in Neuroblastoma: Deep Learning and Topological Data Analysis in Digital Pathology

“…More recently, although alternative approaches are actively pursued [51][52][53][54], well consolidated methodologies derived by CNNs are still being used [55]. In particular, two families of algorithms deserve a mention for the rather large popularity gained in the last few years, both stemming from the original R-CNN model [56].…”

DeepLIIF: An Online Platform for Quantification of Clinical Pathology Slides

“…There are several open source tools for segmenting cells in pathology slides stained with H&E (Hematoxylin & Eosin), IHC (Hematoxylin + brown DAB substrate), and multiplex [5]. These tools are available as (1) web apps (easiest to run with no prerequisite computational expertise), (2) plugins for image analysis toolboxes (mandates familiarity with these toolboxes), (3) coding notebooks (requires basic coding skills), and (4) code-based pipelines (requires significant coding expertise). In this demonstration paper, we focus on tools that are available as web apps and hence are accessible to a broad audience, including both image analysis experts and non-experts.…”

“…This is not the case with the research multiplex immunofluorescence platforms, for example, that output each marker as an independent channel that can be visualized and analyzed separately or as composites. Leveraging this insight, we developed DeepLIIF (published in Nature Machine Intelligence [1]) for virtual multiplex immunofluorescence restaining of standard IHC slides that performs stain deconvolution and cell segmentation/classification in a single step to output clinically relevant IHC scores (mostly quantified as relevant DAB brown cells divided by the total cells). We showed that DeepLIIF, trained on co-registered IHC and multiplex immunofluorescence images, not only achieves state-of-the-art results in IHC nuclear protein marker scoring (Ki67, ER, PR, P53) but also works out-of-the-box for H&E nuclei segmentation as well as cytoplasmic markers (that are expressed close to the nuclei, e.g.…”

“…This demonstration paper accompanies our Nature Machine Intelligence manuscript [1]. Specifically, here we present the DeepLIIF cloud-native platform which supports (1) more than 150 proprietary/non-proprietary input formats via the Bio-Formats standard, (2) interactive adjustment, visualization, and downloading of the IHC quantification results and the accompanying restained images, (3) consumption of an exposed workflow API programmatically or through interactive plugins for open source whole slide image viewers such as QuPath/ImageJ, and (4) auto scaling to automatically and efficiently scale GPU resources based on user demand.…”