RadFusion: Benchmarking Performance and Fairness for Multimodal Pulmonary Embolism Detection from CT and EHR

Abstract

Despite the routine use of electronic health record (EHR) data by radiologists to contextualize clinical history and inform image interpretation, the majority of deep learning architectures for medical imaging are unimodal, i.e., they only learn features from pixel-level information. Recent research revealing how race can be recovered from pixel data alone highlights the potential for serious biases in models which fail to account for demographics and other key patient attributes. Yet the lack of imaging datasets which capture clinical context, inclusive of demographics and longitudinal medical history, has left multimodal medical imaging underexplored. To better assess these challenges, we present RadFusion, a multimodal, benchmark dataset of 1794 patients with corresponding EHR data and high-resolution computed tomography (CT) scans labeled for pulmonary embolisms. We evaluate several representative multimodal fusion models and benchmark their fairness properties across protected subgroups, e.g., gender, race/ethnicity, age. Our results suggest that integrating imaging and EHR data can improve classification performance and robustness without introducing large disparities in the true positive rate between population groups.

Addressing fairness in artificial intelligence for medical imaging

“…Indeed, the first issue one encounters is that a large number of candidate measures exist. One can for instance evaluate fairness by comparing standard ML performance metrics across different sub-groups, such as accuracy 10,[12][13][14][15][16] , or AUC ROC (the area under the receiver operating characteristic curve) [8][9][10][14][15][16][17][18][19][20][21][22] , among others. Alternatively, one can choose to employ one of the (no less than ten) different fairnessspecific criteria formulated by the community 23 in order to audit the presence of bias in a given model 16,18 .…”

“…One can for instance evaluate fairness by comparing standard ML performance metrics across different sub-groups, such as accuracy 10,[12][13][14][15][16] , or AUC ROC (the area under the receiver operating characteristic curve) [8][9][10][14][15][16][17][18][19][20][21][22] , among others. Alternatively, one can choose to employ one of the (no less than ten) different fairnessspecific criteria formulated by the community 23 in order to audit the presence of bias in a given model 16,18 . To complicate matters further, even if one carries out a multi-dimensional study by simultaneously employing multiple metrics 9,10,[14][15][16]20,21,24 , which model to select at the end in a given setting might be no trivial matter and additional information will in general be required.…”

Fair Machine Learning in Healthcare: A Review

“…The multiple perspectives of medical data from different modalities provide information on patient treatment, allowing multimodal models to gradually show their unique advantages in the healthcare field [54]. The research on the fairness issue of multimodal models is not many at current stage [99,17]. A preliminary work presents a multimodal benchmark dataset consisting of 1794 patients and their corresponding EHR data and high-resolution computed tomography (CT) data, called RadFusion [99].…”

“…The research on the fairness issue of multimodal models is not many at current stage [99,17]. A preliminary work presents a multimodal benchmark dataset consisting of 1794 patients and their corresponding EHR data and high-resolution computed tomography (CT) data, called RadFusion [99]. The authors evaluate the performance of several representative multimodal fusion models on a diagnostic task of pulmonary embolism and benchmark their fairness in a protected subgroup.…”

Improving the Fairness of Chest X-ray Classifiers

“…As machine learning models become increasingly integrated in the healthcare setting, one primary concern is whether such models are being used in a fair and ethical way (Ahmad et al, 2020;Wawira Gichoya et al, 2021;. In the field of machine learning for medical imaging, there have been several prior works that benchmark the degree of disparities between protected groups for machine learning mod- (Kinyanjui et al, 2020) and CT scans (Zhou et al, 2021), though to our knowledge, our work is the first to benchmark algorithms for bias reduction in the medical imaging setting.…”