AI for radiographic COVID-19 detection selects shortcuts over signal

Abstract

Artificial intelligence (AI) researchers and radiologists have recently reported AI systems that accurately detect COVID-19 in chest radiographs. However, the robustness of these systems remains unclear. Using state-of-the-art techniques in explainable AI, we demonstrate that recent deep learning systems to detect COVID-19 from chest radiographs rely on confounding factors rather than medical pathology, creating an alarming situation in which the systems appear accurate, but fail when tested in new hospitals. We observe that the approach to obtain training data for these AI systems introduces a nearly ideal scenario for AI to learn these spurious 'shortcuts'. Because this approach to data collection has also been used to obtain training data for the detection of COVID-19 in computed tomography scans and for medical imaging tasks related to other diseases, our study reveals a far-reaching problem in medical-imaging AI. In addition, we show that evaluation of a model on external data is insufficient to ensure AI systems rely on medically relevant pathology, because the undesired 'shortcuts' learned by AI systems may not impair performance in new hospitals. These findings demonstrate that explainable AI should be seen as a prerequisite to clinical deployment of machine-learning healthcare models.

Risk Prediction of Cardiovascular Events by Exploration of Molecular Data with Explainable Artificial Intelligence

“…The problem with lacking insight into model decisions appears when a model learns to predict accurately based on irrelevant features and thus generalizes poorly to other datasets. A recent study, for instance, reported that an accurate AI model trained to identify COVID-19 in chest radiographs actually failed to make use of the relevant information in the images [157]. Due to a consistent patient positioning during imaging, the model instead recognized COVID-19-positive patients based on their shoulder regions.…”

“…Moreover, by knowing the decision pattern, the clinician would be able to assess faithfulness of the prediction. For this reason, explainability is suggested to be an ethical requirement for future clinical decision systems [157]. Here, we introduce different explainable AI methods and give a brief overview of applications in medicine and analysis of molecular pathways.…”

PlayMolecule Glimpse: Understanding Protein–Ligand Property Predictions with Interpretable Neural Networks

“… 16 Learning these shortcuts instead of the underlying nature of the problem is a topic of concern in the field. 17 It is then comprehensible for many machine learning methods to spark criticism regarding the difficulty to understand the rationale behind their predictions. It has been questioned whether a pharmaceutical company would promote a given molecule into a portfolio based only on an opaque prediction made by a neural network, without any clear explanation to support it.…”

Automated COVID-19 diagnosis and prognosis with medical imaging and who is publishing: a systematic review

“…However, the field has inspired controversy. DeGrave et al [ 8 ] demonstrated that combining data from multiple sources, in particular where data from different classes have different acquisition and pre-processing parameters, led to a significant bias that artificially improved the measured performance in many studies. Garcia Santa Cruz et al [ 9 ] presented a review of public CXR datasets, concluding that the most popular datasets used in the literature were at a high risk of introducing bias into reported results.…”