Disparities in Dermatology AI Performance on a Diverse, Curated Clinical Image Set

Identifying the Context Shift between Test Benchmarks and Production Data

“…This list of dimensions largely depends on the context and the degree to which data are subjective, representative, and missing (Mullainathan & Obermeyer, 2017). Recent examples of important contextual dimensions on machine learning tasks include skin color in face recognition (Buolamwini & Gebru, 2018) and dermatology diagnosis (Groh et al, 2021;Daneshjou et al, 2022), background scenery for affect recognition (Kosti et al, 2019), number of people in a video for deepfake detection (Groh et al, 2022a), number of chronic illnesses for algorithmic healthcare risk prediction (Obermeyer et al, 2019), data artifacts like surgical markings (Winkler et al, 2019) or clinically irrelevant labels (Oakden-Rayner et al, 2020) for medical diagnosis classification, and patients' self reports of pain for quantifying severity of knee osteoarthritis (Pierson et al, 2021). Helpful questions that may guide the identification of potential context shifts in complex, human-centered machine learning applications include (and are not limited to): who are represented in the data and as annotators of the data, when and where is the data collected, how do social, geographical, temporal, technological, aesthetic, financial incentives and other idiosyncrasies influence the creation of the data, and why the data is curated as it is.…”