Benchmarking artificial intelligence methods for end-to-end computational pathology

Abstract

Artificial intelligence (AI) can extract subtle visual information from digitized histopathology slides and yield scientific insight on genotype-phenotype interactions as well as clinically actionable recommendations. Classical weakly supervised pipelines use an end-to-end approach with residual neural networks (ResNets), modern convolutional neural networks such as EfficientNet, or non-convolutional architectures such as vision transformers (ViT). In addition, multiple-instance learning (MIL) and clustering-constrained attention MIL (CLAM) are being used for pathology image analysis. However, it is unclear how these different approaches perform relative to each other. Here, we implement and systematically compare all five methods in six clinically relevant end-to-end prediction tasks using data from N=4848 patients with rigorous external validation. We show that histological tumor subtyping of renal cell carcinoma is an easy task which approaches successfully solved with an area under the receiver operating curve (AUROC) of above 0.9 without any significant differences between approaches. In contrast, we report significant performance differences for mutation prediction in colorectal, gastric and bladder cancer. Weakly supervised ResNet- and ViT-based workflows significantly outperformed other methods, in particular MIL and CLAM for mutation prediction. As a reason for this higher performance we identify the ability of ResNet and ViT to assign high prediction scores to highly informative image regions with plausible histopathological image features. We make all source codes publicly available at https://github.com/KatherLab/HIA, allowing easy application of all methods on any end-to-end problem in computational pathology.

DeepMed: A unified, modular pipeline for end-to-end deep learning in computational pathology

“…DeepMed is a straightforward, scalable, and powerful implementation of end-to-end weakly supervised Deep Learning in histology, as demonstrated here. New technologies, such as multiple instance learning 32 and vision transformers 12 , have recently been investigated for such weakly supervised prediction challenges. Although there is limited data on these strategies' realworld performance, some of them may become the de-facto state of the art in the future.…”

“…For example, multiple analysis pipelines have been developed between 2018 and 2021 to predict the mutations of oncogenic driver genes from H&E WSI. 3,5,[9][10][11][12] The overall design of these pipelines is largely identical: they load a WSI, tessellate it into tiles, perform data augmentation and/or normalization, train a CNN, deploy the network on tiles from test patients and use an aggregation function to pool the tile-level predictions on a patient level. 9…”

Swarm learning for decentralized artificial intelligence in cancer histopathology

“…For prediction of molecular features from image data, we adapted our weakly-supervised prediction pipeline "Histology Image Analysis (HIA)" 9 which was demonstrated to outperform similar approaches for mutation prediction in a recent benchmark study. 37 Briefly, the workflow entails the following steps: As preprocessing step, high resolution WSIs were tessellated into patches of size ሺ512 ൈ 512 ൈ 3ሻ pixels and color-normalized. 38 During this process, blurry patches and patches with no tissue are removed from the data set using canny edge detection in OpenCV.…”

“…38 During this process, blurry patches and patches with no tissue are removed from the data set using canny edge detection in OpenCV. 37 Subsequently, we used ResNet18 to extract a ሺ512 ൈ 1ሻ feature vector from 150 randomly selected patches for each patient, as previous work showed that 150 patches are sufficient to obtain robust predictions. 9 Feature vectors and patient-wise target labels (BRAF or MSI status) served as input to a fully connected classification network (FCN).…”

Adversarial attacks and adversarial robustness in computational pathology

“…To date, only very few studies have investigated the use of ViTs in computational pathology. 23,24 Technical studies have described improved robustness of ViTs to adversarial changes to the input data, but this has not been explored in medical applications. 25–27…”