Deep Learning-Based Prediction of Molecular Tumor Biomarkers from H&E: A Practical Review

Abstract

Molecular and genomic properties are critical in selecting cancer treatments to target individual tumors, particularly for immunotherapy. However, the methods to assess such properties are expensive, time-consuming, and often not routinely performed. Applying machine learning to H&E images can provide a more cost-effective screening method. Dozens of studies over the last few years have demonstrated that a variety of molecular biomarkers can be predicted from H&E alone using the advancements of deep learning: molecular alterations, genomic subtypes, protein biomarkers, and even the presence of viruses. This article reviews the diverse applications across cancer types and the methodology to train and validate these models on whole slide images. From bottom-up to pathologist-driven to hybrid approaches, the leading trends include a variety of weakly supervised deep learning-based approaches, as well as mechanisms for training strongly supervised models in select situations. While results of these algorithms look promising, some challenges still persist, including small training sets, rigorous validation, and model explainability. Biomarker prediction models may yield a screening method to determine when to run molecular tests or an alternative when molecular tests are not possible. They also create new opportunities in quantifying intratumoral heterogeneity and predicting patient outcomes.

Deep learning-based risk stratification of preoperative breast biopsies using digital whole slide images

“…In particular, Wang et al obtained an AUC of 0.907 for DeepGrade in their external data regarding resected tumours which is in line with the accuracy we obtained on the biopsy specimen when comparing to resected tumours of NHG1 versus NHG3 (0.908) [ 23 ]. Others who have predicted grade into two groups (low-grade and high-grade) on resected tumour specimens obtained agreements around 80%, and kappa values between 0.59 and 0.64 [ 35 , 37 ]. Despite predicting the resected specimen grade using only biopsy material, we achieved high performance results among NHG1 and NHG3 tumours with an agreement of 82% and a kappa value of 0.65 between biopsy DeepGrade risk groups and pathologist-assigned NHG on resected tumours.…”

Development and deployment of a histopathology-based deep learning algorithm for patient prescreening in a clinical trial

“…While our algorithm was initially designed as a multi-instance learning attention-based network with a CNN backbone 39 , the rapid evolution of technological advancements in computer vision methodologies has outpaced the time required for proper development, validation, deployment in a global clinical study, and drafting this manuscript to share our experience with the scientific community. Notably, newer methods such as vision transformer networks have emerged as alternatives to CNNs 40 , with the potential to offer increased performance, especially when trained on smaller datasets 41 . Additionally, self-supervised learning (SSL) has also shown promising results in the field of histopathology, enabling models to be more generalizable across scanners, staining procedures, and tissue types 42,43 .…”

Computational pathology to improve biomarker testing in breast cancer: how close are we?

“…The precise and reproducible characterization of these biomarkers is essential for patients’ optimal clinical management; however, the traditional technologies [e.g. immunohistochemistry, real-time-PCR, and next-generation sequencing (NGS)] for testing these biomarkers may be troubled by various challenges, including inter-observer/inter-platform variability and reproducibility (Couture, 2022). Effective remedies continue to be subtle due to the extremely complex nature of this crucial task in pathology (Bera et al , 2019; Pisapia et al ., 2022).…”