Artificial Intelligence for the Classification of Pigmented Skin Lesions in Populations with Skin of Color: A Systematic Review

Abstract

Background While skin cancers are less prevalent in people with skin of color, they are more often diagnosed at later stages and have a poorer prognosis. The use of artificial intelligence (AI) models can potentially improve early detection of skin cancers, however the lack of skin color diversity in training datasets may only widen the pre-existing racial discrepancies in dermatology. Objective To systematically review the technique, quality, accuracy, and implications of studies using AI models trained or tested in populations with skin of color, for classification of pigmented skin lesions. Methods PubMed was used to identify any studies describing AI models for classification of pigmented skin lesions. Only studies that used training datasets with at least 10% of images from people with skin of color were eligible. Outcomes on study population, design of AI model, accuracy, and quality of the studies were reviewed. Results Twenty-two eligible articles were identified. Majority of studies were trained on datasets obtained from Chinese (7/22), Korean (5/22), and Japanese populations (3/22). Seven studies used diverse datasets containing Fitzpatrick skin type I-III in combination with at least 10% from Black American, Native American, Pacific Islander or Fitzpatrick IV-VI. AI models producing binary outcomes (e.g., benign vs malignant) reported an accuracy ranging from 70% to 99.7%. Accuracy of AI models reporting multiclass outcomes (e.g., specific lesion diagnosis) was lower, ranging from 43% to 93%. Reader studies, where dermatologists’ classification is compared with AI model outcomes, reported similar accuracy in one study, higher AI accuracy in three studies, and higher clinician accuracy in two studies. A quality review revealed that dataset description and variety, benchmarking, public evaluation, and healthcare application were frequently not addressed. Conclusions While this review provides promising evidence of accurate AI models in skin of color populations, there are still large discrepancies remain in the number of AI models developed in populations with skin of color (particularly Fitzpatrick type IV-VI) and those with largely European ancestry. A lack of publicly available datasets from diverse populations is likely a contributing factor, as is the inadequate reporting of patient-level metadata relating to skin color in training datasets.

Technology innovation to reduce health inequality in skin diagnosis and to improve patient outcomes for people of color: a thematic literature review and future research agenda

“… Jinnai et al (2020) study did not provide a breakdown in the number of Brown and Black participants from each FST group, which is key as a limited number of FST VI and a higher number of IV will affect its validity. Additionally, Liu and Primiero (2023) systematic review predominantly consisted of papers with participants of East Asian origin with some studies containing only 10% of participants with FST type IV–VI. Schakermann et al (2024) study developed the Health Equity Assessment of machine Learning (HEAL) framework to assess the performance of health AI in a case study.…”

“…Similar results are seen in Chen et al ( 2016) study using images of different ethnicities to assess AI performance in identifying melanoma; sensitivity, and specificity results of 90% and 91% were reported. Liu et al (2020) study for Google Health produced results of 'top-1 accuracy' of 71% and 'top-1 sensitivity' of 58% when diagnosing a range of contrasting skin conditions across different skin tones varying from FST I -V. Furthermore, Liu and Primiero (2023) systematic review presented evidence of accurate AI programs for POC within multiple studies showing accuracy levels from 70% to almost 100%.…”

Intelligent Healthcare Platform for Diagnosis of Scalp and Hair Disorders

“…However, they have some potential concerns in clinical practice. AI model performance significantly depends on data quality [14,36,38]. If the dataset is too small and includes inaccurate annotation or inequities, there is a risk of producing incorrect or biased results [14,33,36].…”

Progress and challenges of artificial intelligence in skin of color

“…The inadequate attention given to dermatologic conditions specific to individuals with SOC in medical education magnifies the absence of awareness and evidence-based data for these particular populations. 3 To enhance AI's effectiveness in dermatology for SOC, several key steps are necessary. First, the reliance on the FST for skin type classification should be reevaluated.…”