Biased data, biased AI: deep networks predict the acquisition site of TCGA images

Abstract

Background Deep learning models applied to healthcare applications including digital pathology have been increasing their scope and importance in recent years. Many of these models have been trained on The Cancer Genome Atlas (TCGA) atlas of digital images, or use it as a validation source. One crucial factor that seems to have been widely ignored is the internal bias that originates from the institutions that contributed WSIs to the TCGA dataset, and its effects on models trained on this dataset. Methods 8,579 paraffin-embedded, hematoxylin and eosin stained, digital slides were selected from the TCGA dataset. More than 140 medical institutions (acquisition sites) contributed to this dataset. Two deep neural networks (DenseNet121 and KimiaNet were used to extract deep features at 20× magnification. DenseNet was pre-trained on non-medical objects. KimiaNet has the same structure but trained for cancer type classification on TCGA images. The extracted deep features were later used to detect each slide’s acquisition site, and also for slide representation in image search. Results DenseNet’s deep features could distinguish acquisition sites with 70% accuracy whereas KimiaNet’s deep features could reveal acquisition sites with more than 86% accuracy. These findings suggest that there are acquisition site specific patterns that could be picked up by deep neural networks. It has also been shown that these medically irrelevant patterns can interfere with other applications of deep learning in digital pathology, namely image search. Summary This study shows that there are acquisition site specific patterns that can be used to identify tissue acquisition sites without any explicit training. Furthermore, it was observed that a model trained for cancer subtype classification has exploited such medically irrelevant patterns to classify cancer types. Digital scanner configuration and noise, tissue stain variation and artifacts, and source site patient demographics are among factors that likely account for the observed bias. Therefore, researchers should be cautious of such bias when using histopathology datasets for developing and training deep networks.

Applied machine learning in hematopathology

“…Regardless of model/algorithmic goals, AI/ML must be evaluated using a diverse range of externally sourced data because deep models have a tendency to learn medically irrelevant shortcuts to achieve their medically relevant goals. For instance, it has been shown that models trained on The Cancer Genome Atlas (TCGA) WSIs for cancer subtype classification learned to distinguish hospitals and medical centers that provided WSIs 59 . Additionally, researchers usually evaluate their models on their own data from their own institution.…”

Digital staining facilitates biomedical microscopy

“…It is also worth noting that implementing a system that enables rapid and consistent imaging, correction, and virtual staining of tissue samples would significantly enhance stain uniformity/repeatability. This is particularly crucial considering the lab-based biases present in extensive and reputable databases, such as the digital image collection of The Cancer Genome Atlas (TCGA) ( Dehkharghanian et al, 2023 ).…”

Artificial Intelligence in Digital Pathology for Bladder Cancer: Hype or Hope? A Systematic Review

“…The widespread use of The Cancer Genome Atlas (TCGA) dataset, as seen in 42% of the studies included in our review, further underscores the importance of addressing dataset biases. Some models trained on TCGA have shown a tendency to recognize specific institutional patterns, which, although not medically relevant, could unintentionally affect model performance [ 88 , 89 ]. Moreover, the lack of cross-validation among different cohorts, potential lab-induced tissue artifacts, and the biases from institutional patterns limit model generalizability and clinical application.…”

“…The widespread use of The Cancer Genome Atlas (TCGA) dataset, as seen in 42% of the studies included in our review, further underscores the importance of addressing dataset biases. Some models trained on TCGA have shown a tendency to recognize specific institutional patterns, which, although not medically relevant, could unintentionally affect model performance [88,89].…”