Deep learning in histopathology: the path to the clinic

Laak, Jeroen van der; Litjens, Geert; Ciompi, Francesco

doi:10.1038/s41591-021-01343-4

Cited by 507 publications

(368 citation statements)

References 133 publications

Supporting

Mentioning

364

Contrasting

Unclassified

Order By: Relevance

“…Second, identifying the multi-marker profile of every cell in the H&E-stained tissue biopsy results in earlier identification of cell phenotypes in relation to their interpretation appearances and tissue distributions in the system's pixel pathway. There are a few examples using singleplex IHC to augment annotation of H&E sections for training AI [reviewed by van der Laak et al ( 2021)], including use of cytokeratin IHC to aid identification of breast (Litjens et al, 2018) or prostate (Bulten et al, 2019) cancer cells, detection of mitoses using phosphohistone H3 (Tellez et al, 2018), and detection of breast cancer cells using cytokeratin and Ki67 IHC (Valkonen et al, 2020), and one recent example using mIF of tumor infiltrating lymphocyte (TIL) markers to predict driver mutations in colon cancer (Bian et al, 2021). In either model, mIF and H&E data could be merged by the scanner or by analysis software after scanning to be rendered for viewing and interpretation.…”

Section: Standardization Of Multiplex Immunofluorescence Workflowsmentioning

confidence: 99%

Tissue Multiplex Analyte Detection in Anatomic Pathology – Pathways to Clinical Implementation

Wharton¹,

Wood²,

Manesse³

et al. 2021

Front. Mol. Biosci.

View full text Add to dashboard Cite

Background: Multiplex tissue analysis has revolutionized our understanding of the tumor microenvironment (TME) with implications for biomarker development and diagnostic testing. Multiplex labeling is used for specific clinical situations, but there remain barriers to expanded use in anatomic pathology practice.Methods: We review immunohistochemistry (IHC) and related assays used to localize molecules in tissues, with reference to United States regulatory and practice landscapes. We review multiplex methods and strategies used in clinical diagnosis and in research, particularly in immuno-oncology. Within the framework of assay design and testing phases, we examine the suitability of multiplex immunofluorescence (mIF) for clinical diagnostic workflows, considering its advantages and challenges to implementation.Results: Multiplex labeling is poised to radically transform pathologic diagnosis because it can answer questions about tissue-level biology and single-cell phenotypes that cannot be addressed with traditional IHC biomarker panels. Widespread implementation will require improved detection chemistry, illustrated by InSituPlex technology (Ultivue, Inc., Cambridge, MA) that allows coregistration of hematoxylin and eosin (H&E) and mIF images, greater standardization and interoperability of workflow and data pipelines to facilitate consistent interpretation by pathologists, and integration of multichannel images into digital pathology whole slide imaging (WSI) systems, including interpretation aided by artificial intelligence (AI). Adoption will also be facilitated by evidence that justifies incorporation into clinical practice, an ability to navigate regulatory pathways, and adequate health care budgets and reimbursement. We expand the brightfield WSI system “pixel pathway” concept to multiplex workflows, suggesting that adoption might be accelerated by data standardization centered on cell phenotypes defined by coexpression of multiple molecules.Conclusion: Multiplex labeling has the potential to complement next generation sequencing in cancer diagnosis by allowing pathologists to visualize and understand every cell in a tissue biopsy slide. Until mIF reagents, digital pathology systems including fluorescence scanners, and data pipelines are standardized, we propose that diagnostic labs will play a crucial role in driving adoption of multiplex tissue diagnostics by using retrospective data from tissue collections as a foundation for laboratory-developed test (LDT) implementation and use in prospective trials as companion diagnostics (CDx).

show abstract

Section: Standardization Of Multiplex Immunofluorescence Workflowsmentioning

confidence: 99%

Tissue Multiplex Analyte Detection in Anatomic Pathology – Pathways to Clinical Implementation

Wharton¹,

Wood²,

Manesse³

et al. 2021

Front. Mol. Biosci.

View full text Add to dashboard Cite

show abstract

“…Deep learning algorithms for classifying and diagnosing lung and colon cancer using histopathology images have become a popular research topic in recent years [39], however, due to a paucity of data, no substantial progress has been achieved so far [40]. Despite the lack of data, a few authors have contributed significantly [41].…”

Section: Related Workmentioning

confidence: 99%

Multi-Input Dual-Stream Capsule Network for Improved Lung and Colon Cancer Classification

Ali

2021

Diagnostics

View full text Add to dashboard Cite

Lung and colon cancers are two of the most common causes of death and morbidity in humans. One of the most important aspects of appropriate treatment is the histopathological diagnosis of such cancers. As a result, the main goal of this study is to use a multi-input capsule network and digital histopathology images to build an enhanced computerized diagnosis system for detecting squamous cell carcinomas and adenocarcinomas of the lungs, as well as adenocarcinomas of the colon. Two convolutional layer blocks are used in the proposed multi-input capsule network. The CLB (Convolutional Layers Block) employs traditional convolutional layers, whereas the SCLB (Separable Convolutional Layers Block) employs separable convolutional layers. The CLB block takes unprocessed histopathology images as input, whereas the SCLB block takes uniquely pre-processed histopathological images. The pre-processing method uses color balancing, gamma correction, image sharpening, and multi-scale fusion as the major processes because histopathology slide images are typically red blue. All three channels (Red, Green, and Blue) are adequately compensated during the color balancing phase. The dual-input technique aids the model’s ability to learn features more effectively. On the benchmark LC25000 dataset, the empirical analysis indicates a significant improvement in classification results. The proposed model provides cutting-edge performance in all classes, with 99.58% overall accuracy for lung and colon abnormalities based on histopathological images.

show abstract

“…It is a process of overlaying two or more images from different sources or different time of the same object to align geometrically (Zitová and Flusser 2003). In radiology, this method is used for overlaying images from different sensors, different equipment, or different time (Fox et al, 2008;Tohka and Toga, 2015). This image registration is sometimes done manually using image viewer software when there are not many images, but for multiple image files, automated methods can be used.…”

Section: Image Registration and Normalizationmentioning

confidence: 99%

“…Many reviews have been published in describing the methods and applications of deep learning in pathological image analysis [see (Janowczyk and Madabhushi 2016;Dimitriou et al, 2019;Serag et al, 2019;Roohi et al, 2020;van der Laak et al, 2021)], however, none of these publications discussed or reviewed on the topic of training datasets preparation for ML/DL, which is the most crucial step in developing a useful model in histopathological image analysis. In addition, it is becoming clear that the tumor immune microenvironment (TIME) plays crucial role in determining cancer progression, metastasis, and response to treatment.…”

Section: Introductionmentioning

confidence: 99%

Deep Learning of Histopathology Images at the Single Cell Level

Lee

Lockhart

Xie

et al. 2021

Front. Artif. Intell.

View full text Add to dashboard Cite

The tumor immune microenvironment (TIME) encompasses many heterogeneous cell types that engage in extensive crosstalk among the cancer, immune, and stromal components. The spatial organization of these different cell types in TIME could be used as biomarkers for predicting drug responses, prognosis and metastasis. Recently, deep learning approaches have been widely used for digital histopathology images for cancer diagnoses and prognoses. Furthermore, some recent approaches have attempted to integrate spatial and molecular omics data to better characterize the TIME. In this review we focus on machine learning-based digital histopathology image analysis methods for characterizing tumor ecosystem. In this review, we will consider three different scales of histopathological analyses that machine learning can operate within: whole slide image (WSI)-level, region of interest (ROI)-level, and cell-level. We will systematically review the various machine learning methods in these three scales with a focus on cell-level analysis. We will provide a perspective of workflow on generating cell-level training data sets using immunohistochemistry markers to “weakly-label” the cell types. We will describe some common steps in the workflow of preparing the data, as well as some limitations of this approach. Finally, we will discuss future opportunities of integrating molecular omics data with digital histopathology images for characterizing tumor ecosystem.

show abstract

Deep learning in histopathology: the path to the clinic

Cited by 507 publications

References 133 publications

Tissue Multiplex Analyte Detection in Anatomic Pathology – Pathways to Clinical Implementation

Tissue Multiplex Analyte Detection in Anatomic Pathology – Pathways to Clinical Implementation

Multi-Input Dual-Stream Capsule Network for Improved Lung and Colon Cancer Classification

Deep Learning of Histopathology Images at the Single Cell Level

Contact Info

Product

Resources

About