CheXclusion: Fairness gaps in deep chest X-ray classifiers

Abstract

Machine learning systems have received much attention recently for their ability to achieve expert-level performance on clinical tasks, particularly in medical imaging. Here, we examine the extent to which state-of-the-art deep learning classifiers trained to yield diagnostic labels from X-ray images are biased with respect to protected attributes. We train convolution neural networks to predict 14 diagnostic labels in three prominent public chest X-ray datasets: MIMIC-CXR, Chest-Xray8, and CheXpert. We then evaluate the TPR disparity -the difference in true positive rates (TPR) andunderdiagnosis rate -the false positive rate of a non-diagnosisamong different protected attributes such as patient sex, age, race, and insurance type. We demonstrate that TPR disparities exist in the state-of-the-art classifiers in all datasets, for all clinical tasks, and all subgroups. We find that TPR disparities are most commonly not significantly correlated with a subgroup's proportional disease burden; further, we find that some subgroups and subsection of the population are chronically underdiagnosed. Such performance disparities have real consequences as models move from papers to products, and should be carefully audited prior to deployment. CCS CONCEPTS• Computing methodologies → Neural networks; Learning latent representations; • Applied computing → Imaging.

An empirical framework for domain generalization in clinical settings

“…Transfer learning [57] addresses this by using a model pretrained on a large-scale dataset and fine-tuning it to the downstream task. This method has been commonly used in designing medical image classifiers [4,37,66,70,80]. In these settings, the deep neural network is initialized with a pretrained model (for example, trained on Ima-geNet [24]) and then are finetuned on downstream medical images.…”

“…In these settings, the deep neural network is initialized with a pretrained model (for example, trained on Ima-geNet [24]) and then are finetuned on downstream medical images. Transfer learning has been shown to be effective at increasing model performance in chest X-ray classifiers [66,70], though there are cases where a model trained from scratch can perform just as well [64].…”

AUC Maximization in the Era of Big Data and AI: A Survey

“…Two issues are of foremost importance, namely fairness [8,11,21,99] and interpretability [4,25,59]. Although these issues have received tremendous attention in the literature for medical image classification [27,127,128], developing fair and interpretable DAM methods remains to be explored. Some outstanding questions and work include (i) how to develop scalable in-processing algorithms for optimizing AUC under AUC-based fairness constraints [13,76]; (ii) how to develop scalable and interpretable DAM methods; (iii) evaluating these fairness-aware and interpretable AUC optimizaiton methods on large-scale medical image datasets.…”

CheXpert++: Approximating the CheXpert labeler for Speed,Differentiability, and Probabilistic Output

“…This work demonstrates both the utility of CheXpert++ as a direct replacement for CheXpert in contexts where speed, differentiability, or probabilistic output matter, but also suggest that perhaps we need to place greater emphasis on strategies to intelligently and efficiently improve the quality of silver-labels in medicine. Many works have already used the CheXpert labels across different datasets, including Liu et al, 2019;Seyyed-Kalantari et al, 2020), among others, despite known issues relating to label quality. Here, we demonstrate first through the power of the apparent inductive biases of CheXpert++, as it improves on CheXpert's labels simply by training to match them, and second through our proof-of-concept active learning study, where we improve performance notably with minimal effort, that simple efforts could've appeased some of these label quality issues.…”