Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Dataset

Abstract

How does the accuracy of deep neural network models trained to classify clinical images of skin conditions vary across skin color? While recent studies demonstrate computer vision models can serve as a useful decision support tool in healthcare and provide dermatologist-level classification on a number of specific tasks, darker skin is underrepresented in the data. Most publicly available data sets do not include Fitzpatrick skin type labels. We annotate 16,577 clinical images sourced from two dermatology atlases with Fitzpatrick skin type labels and open-source these annotations. Based on these labels, we find that there are significantly more images of light skin types than dark skin types in this dataset. We train a deep neural network model to classify 114 skin conditions and find that the model is most accurate on skin types similar to those it was trained on. In addition, we evaluate how an algorithmic approach to identifying skin tones, individual typology angle, compares with Fitzpatrick skin type labels annotated by a team of human labelers.

Detecting Melanoma Fairly: Skin Tone Detection and Debiasing for Skin Lesion Classification

“…Groh et al [10] illustrate that CNNs perform better at classifying images with similar skin tones to those the model was trained on. Performance is, therefore, likely to be poor for patients with darker skin tones when the training data is predominantly images of light-skinned patients, which is the case with many of the current commonly-used dermoscopic training datasets such as the ISIC archive data [18,6].…”

“…We calculate the individual typology angle (ITA) of the healthy skin in each image to approximate skin tone [15,10], given by:…”

“…where L and b are obtained by converting RGB pixel values to the CIELab colour space. We propose a simpler and more efficient method for isolating healthy skin than the segmentation method used in [15,10]. Across all skin tones, lesions and blemishes are mostly darker than the surrounding skin.…”

“…Consequently, to select a non-diseased patch of skin, we take 8 samples of 20×20 pixels from around the edges of each image and use the sample with the highest ITA value (lightest skin tone) as the estimated skin tone. We map the ITA values to Fitzpatrick skin types using the thresholds from [10]. We pre-process each image using blackhat morphology to remove hair, preventing dark pixels from hairs skewing the calculation.…”

“…For example, an issue that is commonly raised in the existing literature is skin tone bias in lesion classification tasks. Groh et al [10] provide a compiled dataset of clinical lesions with human annotated Fitzpatrick skin type [8] labels, and show that CNNs perform best at classifying skin types similar to the skin types in the training data used. We use the skin type labels in this dataset as the target for supervised bias removal methods to evaluate their effectiveness in helping melanoma classification models generalise to unseen skin types.…”