Towards a general-purpose foundation model for computational pathology

Chen, Richard J.; Ding, Tong; Lu, Ming Y.; Williamson, Drew F. K.; Jaume, Guillaume; Song, Andrew H.; Chen, Bowen; Zhang, Andrew; Shao, Daniel; Shaban, Muhammad; Williams, Mane; Oldenburg, Lukas; Weishaupt, Luca L.; Wang, Judy J.; Vaidya, Anurag; Le, Long Phi; Gerber, Georg; Sahai, Sharifa; Williams, Walt; Mahmood, Faisal

doi:10.1038/s41591-024-02857-3

Cited by 41 publications

(10 citation statements)

References 145 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The HistoTME model consists of two main components: a frozen feature extraction component and a trainable attention-based multiple instance learning (AB-MIL) component 33 . In our efforts to efficiently train HistoTME, we explored three state-of-the-art open-source foundational models— CTransPath, RetCCL, and UNI 25,26,30 —as potential feature extractors. Additionally, we conducted experiments with two distinct approaches for AB-MIL: a single-task approach featuring a unique attention and multilayer perceptron (MLP) head for each TME signature, and a multitask approach, which incorporates a shared attention head for functionally related TME signatures but maintains separate MLP heads for each individual TME signature (Fig.…”

Section: Resultsmentioning

confidence: 99%

“…Due to the limited benchmarking of foundation models for continuous biomarker prediction tasks, we experimented with three popular foundation models as feature extractors. We found out that the UNI foundation model 25 , when paired with multi-task AB-MIL, achieved the best predictive performance for the various TME signature prediction tasks. This improved performance likely stems from the considerably large histopathology datasets used to pre-train the UNI foundation model, as well as the added regularization induced by multitask learning.…”

Section: Discussionmentioning

confidence: 98%

“…While these approaches are extremely valuable, they are limited by availability of relevant pixel-level annotations from expert pathologists, which are time and resource consuming to generate. To overcome these limitations, several research groups have alternatively proposed the use of weakly-supervised deep learning models, which can be trained to perform various downstream computational pathology tasks such as tumor subtyping and prognosis 25–30 without any region or pixel-level annotations.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Predicting the Tumor Microenvironment Composition and Immunotherapy Response in Non-Small Cell Lung Cancer from Digital Histopathology Images

Patkar,

Chen,

Basnet

et al. 2024

Preprint

View full text Add to dashboard Cite

Immune checkpoint inhibitors (ICI) have become integral to treatment of non-small cell lung cancer (NSCLC). However, reliable biomarkers predictive of immunotherapy efficacy are limited. Here, we introduce HistoTME, a novel weakly supervised deep learning approach to infer the tumor microenvironment (TME) composition directly from histopathology images of NSCLC patients. We show that HistoTME accurately predicts the expression of 30 distinct cell type-specific molecular signatures directly from whole slide images, achieving an average Pearson correlation of 0.5 with the ground truth on independent tumor cohorts. Furthermore, we find that HistoTME-predicted microenvironment signatures and their underlying interactions improve prognostication of lung cancer patients receiving immunotherapy, achieving an AUROC of 0.75[95% CI: 0.61-0.88] for predicting treatment responses following first-line ICI treatment, utilizing an external clinical cohort of 652 patients. Collectively, HistoTME presents an effective approach for interrogating the TME and predicting ICI response, complementing PD-L1 expression, and bringing us closer to personalized immuno-oncology.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 98%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Predicting the Tumor Microenvironment Composition and Immunotherapy Response in Non-Small Cell Lung Cancer from Digital Histopathology Images

Patkar,

Chen,

Basnet

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…Second, it remains challenging to design a model architecture that can effectively capture both local patterns in individual tiles and global patterns across whole slides 35 – 39 . Existing models often treat each image tile as an independent sample and formulate slide-level modelling as multiple instance learning 4 , 40 – 43 , thus limiting their ability to model complex global patterns in gigapixel whole slides. A notable exception is Hierarchical Image Pyramid Transformer (HIPT), which explores hierarchical self-attention over the tiles 35 .…”

Section: Mainmentioning

confidence: 99%

A whole-slide foundation model for digital pathology from real-world data

Xu,

Usuyama,

Bagga

et al. 2024

Nature

View full text Add to dashboard Cite

Digital pathology poses unique computational challenges, as a standard gigapixel slide may comprise tens of thousands of image tiles1–3. Prior models have often resorted to subsampling a small portion of tiles for each slide, thus missing the important slide-level context4. Here we present Prov-GigaPath, a whole-slide pathology foundation model pretrained on 1.3 billion 256 × 256 pathology image tiles in 171,189 whole slides from Providence, a large US health network comprising 28 cancer centres. The slides originated from more than 30,000 patients covering 31 major tissue types. To pretrain Prov-GigaPath, we propose GigaPath, a novel vision transformer architecture for pretraining gigapixel pathology slides. To scale GigaPath for slide-level learning with tens of thousands of image tiles, GigaPath adapts the newly developed LongNet5 method to digital pathology. To evaluate Prov-GigaPath, we construct a digital pathology benchmark comprising 9 cancer subtyping tasks and 17 pathomics tasks, using both Providence and TCGA data6. With large-scale pretraining and ultra-large-context modelling, Prov-GigaPath attains state-of-the-art performance on 25 out of 26 tasks, with significant improvement over the second-best method on 18 tasks. We further demonstrate the potential of Prov-GigaPath on vision–language pretraining for pathology7,8 by incorporating the pathology reports. In sum, Prov-GigaPath is an open-weight foundation model that achieves state-of-the-art performance on various digital pathology tasks, demonstrating the importance of real-world data and whole-slide modelling.

show abstract

“…Recently developed vision models allow unsupervised extraction of morphological features which can then be used for clustering and data integration tasks. During the hackathon general purpose models trained on imagenet and UNI (Chen et al, 2024) a model specifically tuned on histopathology were evaluated.…”

Section: Workgroup Spatial Multi-omicsmentioning

confidence: 99%

VIB Hackathon on spatial omics tools and methods

Rombaut,

Pollaris,

Sang-aram

et al. 2024

Preprint

View full text Add to dashboard Cite

During a three-day hackathon, work was performed on various topics within the field of spatial omics data analysis. The topics were organized in five workgroups and included benchmarking, pipelines, spatial transcriptomics, spatial proteomics, spatial multi-omics and cell-cell communication. Most tools and methods were considered in the context of the Python ecosystem for spatial (SpatialData) and single-cell (scverse) data analysis.

show abstract

Towards a general-purpose foundation model for computational pathology

Cited by 41 publications

References 145 publications

Predicting the Tumor Microenvironment Composition and Immunotherapy Response in Non-Small Cell Lung Cancer from Digital Histopathology Images

Predicting the Tumor Microenvironment Composition and Immunotherapy Response in Non-Small Cell Lung Cancer from Digital Histopathology Images

A whole-slide foundation model for digital pathology from real-world data

VIB Hackathon on spatial omics tools and methods

Contact Info

Product

Resources

About