Hist2Cell: Deciphering Fine-grained Cellular Architectures from Histology Images

Abstract

Histology images, with low cost, are unleashing great power of predicting cellular phenotypes in tissue, thanks to the emerging spatial transcriptomics serving as annotations. Recent efforts aimed to predict individual gene expression, suffering from low accuracy and high variability, while no methods are tailored to predict cell types - the most critical phenotype. Here, we present Hist2Cell, a Vision Graph- Transformer framework, to resolve fine-grained cell types directly from histology images and further create cellular maps of diverse tissues at a customizable resolution. Specifically, trained on human lung and breast cancer spatial transcriptome datasets, Hist2Cell accurately predicts the abundance of each cell type across space, effectively capturing their colocalization directly from histology images. Moreover, without the need for model re-training, it robustly generalizes to large-scale histology cohorts of breast cancer samples from TCGA, highlighting recurrent cell type colocalization. Therefore, Hist2Cell enables cost-efficient histology analysis for large-scale studies of spatial biology and clinical diagnostics.