Scaling Self-Supervised Learning for Histopathology with Masked Image Modeling

Abstract

Computational pathology is revolutionizing the field of pathology by integrating advanced computer vision and machine learning technologies into diagnostic workflows. Recently, self-supervised learning (SSL) has emerged as a promising solution to learn representations from histology patches, leveraging large volumes of unannotated whole slide images (WSI). In particular, Masked Image Modeling (MIM) showed remarkable results and robustness over purely contrastive learning methods. In this work, we explore the application of MIM to histology using iBOT, a self-supervised transformer-based framework. Through a wide range of downstream tasks over seven cancer indications, we provide recommendations on the pre-training of large models for histology data using MIM. First, we demonstrate that in-domain pre-training with iBOT outperforms both ImageNet pre-training and a model pre-trained with a purely contrastive learning objective, MoCo V2. Second, we show that Vision Transformers models (ViT), when scaled appropriately, have the capability to learn pan-cancer representations that benefit a large variety of downstream tasks. Finally, our iBOT ViT-Base model, pre-trained on more than 40 million histology images from 16 different cancer types, achieves state-of-the-art performance in most weakly-supervised WSI classification tasks compared to other SSL frameworks.

Democratizing computational pathology: optimized Whole Slide Image representations for The Cancer Genome Atlas

“…Each module can be updated independently, allowing the entire framework to continuously benefit from advancements in their respective domains. We illustrate this by comparing various pre-trained tile-encoders as shown in Supplementary Table 1, notably the recent ctranspath network, and demonstrate that the WSI Giga-SSL representation benefits from an improved tile-encoder; a research effort that already receives much attention [15–17].…”

Robust Evaluation of Deep Learning-based Representation Methods for Survival and Gene Essentiality Prediction on Bulk RNA-seq Data

“…We focused on building a robust, reproducible and fair benchmarking framework to compare different machine learning models built on bulk RNA-seq data with respect to their performance on different downstream tasks. The key elements used in the pipeline we developed (repeated holdout cross-validation and fixed HP tuning budget) are not new per-se (Filiot et al 2023) but to our knowledge this framework has been used in a limited number of studies (Smith et al 2020) evaluating ML models on omics data in cancer-specific downstream tasks. Tuning directly on a cross-validation without a held out test set could lead to inflated performance and justifies the need for a nested cross-validation framework or the proposed repeated holdout test set.…”

Multiplexed tumor profiling with generative AI accelerates histopathology workflows and improves clinical predictions

“…To reach its full potential, future work will be crucial to establish the use of the VirtualMultiplexer in real-world settings. From a technical standpoint, the generated virtual multiplexed stainings can enable the development of foundational models for IHC, as they have been successfully developed for brightfield H&E images [54][55][56]. However, developing such models requires high volumes of data, which is potentially challenging to acquire for IHC.…”