Deep learning models for COVID-19 chest x-ray classification: Preventing shortcut learning using feature disentanglement

Abstract

In response to the COVID-19 global pandemic, recent research has proposed creating deep learning based models that use chest radiographs (CXRs) in a variety of clinical tasks to help manage the crisis. However, the size of existing datasets of CXRs from COVID-19+ patients are relatively small, and researchers often pool CXR data from multiple sources, for example, using different x-ray machines in various patient populations under different clinical scenarios. Deep learning models trained on such datasets have been shown to overfit to erroneous features instead of learning pulmonary characteristics -- a phenomenon known as shortcut learning. We propose adding feature disentanglement to the training process, forcing the models to identify pulmonary features from the images while penalizing them for learning features that can discriminate between the original datasets that the images come from. We find that models trained in this way indeed have better generalization performance on unseen data; in the best case we found that it improved AUC by 0.13 on held out data. We further find that this outperforms masking out non-lung parts of the CXRs and performing histogram equalization, both of which are recently proposed methods for removing biases in CXR datasets.

Public Covid-19 X-ray datasets and their impact on model bias – A systematic review of a significant problem

“…But the most general risk to all such compilations is confounding by dataset identity. This effect was for example clearly demonstrated for the COVIDx dataset by means of an external test set in Robinson et al. (2021) and more general for different COVID-19 related dataset merges in Ahmed, Goldgof, Paul, Goldgof, Hall, 2021 , Ahmed, Hall, Goldgof, Goldgof, Paul .…”

“…But many authors utilise small private dataset together with larger public datasets. An exemplary case is the use of private datasets as external test data, therewith assessing the transportability of models trained on (merged) public datasets to the private test data population and thus to the underlying hospital setting ( Park, Kim, Oh, Seo, Lee, Kim, Moon, Lim, Ye , Kim, Park, Oh, Seo, Lee, Kim, Moon, Lim, Ye , Elgendi, Nasir, Tang, Smith, Grenier, Batte, Spieler, Leslie, Menon, Fletcher, et al., 2021 , Robinson, Trivedi, Blazes, Ortiz, Desbiens, Gupta, Dodhia, Bhatraju, Liles, Lee, et al., 2021 ).…”

“…Another mitigation strategy, special to deep learning, is the adversarial training of models to explicitly ignore confounding variables ( Zhao et al., 2020 ). This approach is a conceptual extension of adversarial domain adaption ( Ganin et al., 2016 ) which has been shown to mitigate some of the dataset identity confounding in models trained on the COVIDx dataset ( Robinson et al., 2021 ). Nevertheless, for all mitigation strategies to work, at least the confounders have to be documented.…”

Augmenting existing deterioration indices with chest radiographs to predict clinical deterioration

“…Augmentation, rather than re-training, requires less data because the augmentation model is trained on the deterioration index output (rather than its inputs), resulting in a smaller feature set during training. When using a large variety of different data sources, multimodal models trained from scratch tend to overfit to limited data [ 19 ]. By using only the outputs of existing validated models, we limit the number of features used in the image-augmented deterioration indices, mitigating the risk of overfitting.…”

Federated Learning for Microvasculature Segmentation and Diabetic Retinopathy Classification of OCT Data

“…This presents a nearly insurmountable hurdle for collaborative data sharing between institutions. Additionally, there exists a possibility that a model trained solely on medical data available in its own 'data island' is significantly overfitted, especially if all the data originates from one source [18]. This is the case for image J o u r n a l P r e -p r o o f processing algorithms, where images originating from one source may have distinct features that may lead to overfitting as training progresses.…”