Prediction of cancer treatment response from histopathology images through imputed transcriptomics

Abstract

Advances in artificial intelligence have paved the way for predicting cancer patients' survival and response to treatment from hematoxylin and eosin (H&E)-stained tumor slides. Extant approaches do so either directly from the H&E images or via prediction of actionable mutations and gene fusions. Here we present the first genetic interactions (GI)-based approach for predicting patient response to treatment, founded on two conceptual steps: (1) First, we build DeepPT, a deep-learning framework that predicts tumor gene expression from H&E slides, and subsequently, (2) we apply ENLIGHT - a previously published GI based approach - to predict patient treatment response from the inferred tumor expression. DeepPT was trained on images and corresponding transcriptomics data of TCGA breast, kidney, lung, and brain tumor samples. Testing DeepPT transcriptomics prediction ability, we find that it generalizes well to predicting the expression of two breast and brain cancer unseen independent datasets. Studying samples from a recently published large multi-omics breast cancer clinical trial, we applied ENLIGHT to the expression predicted by DeepPT from the tumor slides. We find that it successfully predicts true responders with a clinically meaningful hazard ratio of about six. These results put forward a general framework for predicting patient response to a broad array of targeted and checkpoint therapies from the histological images. If corroborated further, the new approach could augment the feasibility of precision oncology in developing countries and in other situations where comprehensive molecular profiling is not available.

Predicting the Tumor Microenvironment Composition and Immunotherapy Response in Non-Small Cell Lung Cancer from Digital Histopathology Images

“…In this work, we introduce HistoTME, a novel weekly supervised deep learning method to characterize the TME of patients from H&E slides, leveraging enhanced feature extraction capabilities of recent digital pathology foundation models. In contrast to recent deep learning approaches, which aim to predict spatially resolved gene expression profiles from histopathology images [42][43][44][45] , our approach aims to infer the TME composition through estimating the expression of distinct functional TME signatures. A key advantage of this approach is that by learning to predict the expression of gene signatures, we not only avoid overfitting to the expression of individual genes but also increase interpretability by directly relating specific histopathological features to previously established biological concepts 32 .…”

Spatial gene expression at single-cell resolution from histology using deep learning with GHIST

“…DeepPT 8 - DeepPT was implemented in Python and PyTorch using hyperparameter values as outlined in the original paper. The ResNet-50 backbone used was pre-trained on ImageNet.…”

“…These methods learn visual and spatial associations between matched H&E images and spot-level SGE data. Existing methods typically examine H&E image patches with a Convolutional Neural Network (CNN) 8,10,12 or Transformer 9,11 backbone. For example, ST-Net leveraged a CNN-based backbone to extract an embedding of image patches that corresponded to spots, then applied a fully connected layer to predict gene expressions of each spot.…”

Benchmarking the translational potential of spatial gene expression prediction from histology

“…HE2RNA 48 and ISG 49 ) were excluded. Although we excluded all methods predicting bulk gene expression from histology, we adapted the architecture from DeepPT 23 for predicting spatial transcriptomics. This was done as the model architecture was relatively clear to reproduce and the authors had also aimed to predict clinical outcomes using predicted SGE from their method, which was one of our evaluation categories of interest.…”

“…Evidently, different evaluation frameworks have been used by different developers to assess various aspects of their methods. For instance, ST-Net 5 , DeepSpaCE 9 and DeepPT 23 evaluated the performance of their models on external The Cancer Genome Atlas (TCGA) data to determine the generalisability of their models. Hist2ST 8 performed an ablation study to assess the contributions of different components of their deep learning architecture.…”