Identifying transcriptomic correlates of histology using deep learning

Abstract

Linking phenotypes to specific gene expression profiles is an extremely important problem in biology, which has been approached mainly by correlation methods or, more fundamentally, by studying the effects of gene perturbations. However, genome-wide perturbations involve extensive experimental efforts, which may be prohibitive for certain organisms. On the other hand, the characterization of the various phenotypes frequently requires an expert’s subjective interpretation, such as a histopathologist’s description of tissue slide images in terms of complex visual features (e.g. ‘acinar structures’). In this paper, we use Deep Learning to eliminate the inherent subjective nature of these visual histological features and link them to genomic data, thus establishing a more precisely quantifiable correlation between transcriptomes and phenotypes. Using a dataset of whole slide images with matching gene expression data from 39 normal tissue types, we first developed a Deep Learning tissue classifier with an accuracy of 94%. Then we searched for genes whose expression correlates with features inferred by the classifier and demonstrate that Deep Learning can automatically derive visual (phenotypical) features that are well correlated with the transcriptome and therefore biologically interpretable. As we are particularly concerned with interpretability and explainability of the inferred histological models, we also develop visualizations of the inferred features and compare them with gene expression patterns determined by immunohistochemistry. This can be viewed as a first step toward bridging the gap between the level of genes and the cellular organization of tissues.

Deep learning features encode interpretable morphologies within histological images

“…We identified clusters of highly correlated mone-gene sets, demonstrating clear connection of mones to the underlying genetics. Some recent studies have used exhaustive sets of deep learning features to predict expression profiles 16 – 18 , but our work shows that small mone-gene clusters can be sufficient and provide simpler interpretability. Both supervised and unsupervised analyses identify meaningful clusters (“ Mones have interpretable correlations with gene expression ” section).…”

“…However, models integrating these diverse modalities are needed. The feasibility of doing so is supported by work establishing the connection between modalities, for example by using CNNs to predict expression values of specific genes from H&E images 16 – 18 . Because of the architectural complexity of CNNs, it has often been assumed that CNN-based decompositions of images into features are not interpretable.…”

Automated image analysis method to detect and quantify fat cell infiltration in hematoxylin and eosin stained human pancreas histology images

“…A sample size of three high resolution pancreas histology images were analyzed with sample IDs GTEX-11DXZ-0826, GTEX-1122O-0726, GTEX-1117F-1726. The images have magnification of 20x (0.4942 mpp-microns per pixel) [13]. The source code was written in Python version 3.8.0 (Python Software Foundation, Beaverton, Oregon, USA) with Core i7 3 rd Generation CPU, 8 GB RAM and Intel HD graphics 4000 GPU.…”

Integrating pathomics with radiomics and genomics for cancer prognosis: A brief review

“…The deep learning model was treated as a “black box” method since the learnt features and model decision making were difficult to explain. Researchers have tried to open this box by using activation maps ( 12 , 38 ) and providing visualization of learnt features ( 39 ). Compared to deep learning approaches, the hand-crafted features extracted from histology image and radiology image provide better explainability since the features were pre-defined, either in a domain agnostic ( 13 , 40 ) or domain inspired ( 8 ) way.…”