Transformer with Convolution and Graph-Node co-embedding: An accurate and interpretable vision backbone for predicting gene expressions from local histopathological image

Abstract

Inferring gene expressions from histopathological images has always been a fascinating but challenging task due to the huge differences between the two modal data. Previous works have used modified DenseNet121 to encode the local images and make gene expression predictions. And later works improved the prediction accuracy of gene expression by incorporating the coordinate information from images and using all spots in the tissue region as input. While these methods were limited in use due to model complexity, large demand on GPU memory, and insufficient encoding of local images, thus the results had low interpretability, relatively low accuracy, and over-smooth prediction of gene expression among neighbor spots. In this paper, we propose TCGN, (Transformer with Convolution and Graph-Node co-embedding method) for gene expression prediction from H&E stained pathological slide images. TCGN consists of convolutional layers, transformer encoders, and graph neural networks, and is the first to integrate these blocks in a general and interpretable computer vision backbone for histopathological image analysis. We trained TCGN and compared its performance with three existing methods on a publicly available spatial transcriptomic dataset. Even in the absence of the coordinates information and neighbor spots, TCGN still outperformed the existing methods by 5% and achieved 10 times higher prediction accuracy than the counterpart model. Besides its higher accuracy, our model is also small enough to be run on a personal computer and does not need complex building graph preprocessing compared to the existing methods. Moreover, TCGN is interpretable in recognizing special cell morphology and cell-cell interactions compared to models using all spots as input that are not interpretable. A more accurate omics information prediction from pathological images not only links genotypes to phenotypes so that we can predict more biomarkers that are expensive to test from histopathological images that are low-cost to obtain, but also provides a theoretical basis for future modeling of multi-modal data. Our results support that TCGN is a useful tool for inferring gene expressions from histopathological images and other potential histopathological image analysis studies.

Benchmarking the translational potential of spatial gene expression prediction from histology

“…As there were several methods which were recently developed, not all methods included publicly available code at the time of investigation (e.g. TCGN 16 , BrST-Net 17 , NSL 15 , TransformerST 18 and STimage 42 ) were excluded as a result. BLEEP 14 was unable to be run due to lack of tutorials, and EGN 11 was run however only the training set produced reasonable correlations and thus was excluded from our results.…”

“…To date, several approaches have been developed to predict in-silico spatially resolved gene expression (SGE) patterns using H&E data alone (Figure 1a) (see Supplementary Table 1) [5][6][7][8][9][10][11][12][13][14][15][16][17][18] . Among these methods, Convolutional Neural Networks (CNN) and Transformers are commonly selected architectures for extracting local and global 2D vision features around each sequenced spot from corresponding histology image patches.…”

“…Among these methods, Convolutional Neural Networks (CNN) and Transformers are commonly selected architectures for extracting local and global 2D vision features around each sequenced spot from corresponding histology image patches. Some methods further implemented extra components, including super-resolution enhancement of SGE 9,18,19 and Graph Neural Networks (GNN) 8,12,16,18 to capture neighbourhood relationships between adjacent spots. These learned features are then used to predict SGE that enable image-based screening for molecular biomarkers with spatial variation.…”