Health equity assessment of machine learning performance (HEAL): a framework and dermatology AI model case study

Schaekermann, Mike; Spitz, Terry; Pyles, Malcolm; Cole-Lewis, Heather; Wulczyn, Ellery; Pfohl, Stephen R.; Martin, Donald; Jaroensri, Ronnachai; Keeling, Geoff; Liu, Yuan; Farquhar, Stephanie; Xue, Qinghan; Lester, Jenna; Hughes, Cían; Strachan, Patricia; Tan, Fraser; Bui, Peggy; Mermel, Craig H.; Peng, Lily H.; Matias, Yossi; Corrado, Greg S.; Webster, Dale R.; Virmani, Sunny; Semturs, Christopher; Liu, Yun; Horn, Ivor; Cameron Chen, Po-Hsuan

doi:10.1016/j.eclinm.2024.102479

Cited by 4 publications

(3 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Additionally, Liu and Primiero (2023) systematic review predominantly consisted of papers with participants of East Asian origin with some studies containing only 10% of participants with FST type IV–VI. Schakermann et al (2024) study developed the Health Equity Assessment of machine Learning (HEAL) framework to assess the performance of health AI in a case study. While Schakermann et al (2024) case was carefully sampled to create a balance in demographics, there was still a poor representation of FST V-VI and American Indian/Alaska Natives.…”

Section: Resultsmentioning

confidence: 99%

“… Schakermann et al (2024) study developed the Health Equity Assessment of machine Learning (HEAL) framework to assess the performance of health AI in a case study. While Schakermann et al (2024) case was carefully sampled to create a balance in demographics, there was still a poor representation of FST V-VI and American Indian/Alaska Natives. These studies’ results are skewed due to poor representation of POC affecting the results generalisability or show the struggle in trying to work with balanced data sets due to limited resources.…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Technology innovation to reduce health inequality in skin diagnosis and to improve patient outcomes for people of color: a thematic literature review and future research agenda

Khatun,

Spinelli,

Colecchia

2024

Front. Artif. Intell.

View full text Add to dashboard Cite

The health inequalities experienced by ethnic minorities have been a persistent and global phenomenon. The diagnosis of different types of skin conditions, e.g., melanoma, among people of color is one of such health domains where misdiagnosis can take place, potentially leading to life-threatening consequences. Although Caucasians are more likely to be diagnosed with melanoma, African Americans are four times more likely to present stage IV melanoma due to delayed diagnosis. It is essential to recognize that additional factors such as socioeconomic status and limited access to healthcare services can be contributing factors. African Americans are also 1.5 times more likely to die from melanoma than Caucasians, with 5-year survival rates for African Americans significantly lower than for Caucasians (72.2% vs. 89.6%). This is a complex problem compounded by several factors: ill-prepared medical practitioners, lack of awareness of melanoma and other skin conditions among people of colour, lack of information and medical resources for practitioners’ continuous development, under-representation of people of colour in research, POC being a notoriously hard to reach group, and ‘whitewashed’ medical school curricula. Whilst digital technology can bring new hope for the reduction of health inequality, the deployment of artificial intelligence in healthcare carries risks that may amplify the health disparities experienced by people of color, whilst digital technology may provide a false sense of participation. For instance, Derm Assist, a skin diagnosis phone application which is under development, has already been criticized for relying on data from a limited number of people of color. This paper focuses on understanding the problem of misdiagnosing skin conditions in people of color and exploring the progress and innovations that have been experimented with, to pave the way to the possible application of big data analytics, artificial intelligence, and user-centred technology to reduce health inequalities among people of color.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Technology innovation to reduce health inequality in skin diagnosis and to improve patient outcomes for people of color: a thematic literature review and future research agenda

Khatun,

Spinelli,

Colecchia

2024

Front. Artif. Intell.

View full text Add to dashboard Cite

show abstract

“…Algorithms developed by the International Skin Imaging Collaboration (ISIC) can match the expertise of professional dermatologists in simulated tests 41,42 . However, data biases in AI training models can affect the universality across different ethnicities and socio-economic populations, leading to algorithmic biases and health disparities to some extent [43][44][45][46] . For example, the diagnostic accuracy of ISIC does not include individuals with Fitzpatrick phototype III or higher 47 .…”

Section: Discussionmentioning

confidence: 99%

Exploring the analysis capabilities and clinical application potential of the Claude 3 Opus in different dermatologic images: the development of a large-scale multimodal model to assist in dermatology clinical practice (Preprint)

Huang,

Liu,

Zhang

et al. 2024

Preprint

View full text Add to dashboard Cite

BACKGROUND Publicly available, accessible, and user-friendly artificial intelligence is expected to serve in medical processes. Claude3, a newly introduced large-scale multimodal model, has demonstrated significantly superior image analysis capabilities compared to other models in official tests. However, there is no research reporting the potential of Claude3 in medical image analysis. OBJECTIVE To explore the analysis capabilities and potential applications of Claude3 Opus on dermatologic images. METHODS Dermoscopy and dermatopathology images from textbooks were collected, and question templates for different types of diseases were designed for Claude3 Opus. Three dermatologists used a structured scoring system with five modules to evaluate Claude3 Opus' analysis of images based on recognition, description, completeness, diagnosis, and clinical application, with each module scored out of 5 and a total score of 25. RESULTS A total of 330 images were collected. Claude3 Opus scored highest in pigmented disorders dermoscopy images (18.65/25), followed by vascular disorders dermoscopy (15.97/25) and vascular disorders dermatopathology images (15.86/25). In pigmented disorders, its score in dermoscopy (18.65/25) was significantly higher than in dermatopathology images (14.54/25), but no such difference existed in vascular disorders. Among the five modules, Claude3 Opus' recognition score (3.65/5) was significantly higher than the other four modules. There was no difference between description score (3.14/5) and completeness score (3.22/5), but they were significantly higher than the diagnostic score (2.47/5). Claude3 Opus scored higher in malignant diseases than benign diseases, regardless of dermoscopy or dermatopathology images (all p-values <0.05), with no impact from different magnifications in dermatopathology images (p>0.05) and no impact from different number of evaluators. CONCLUSIONS Claude3 Opus exhibits strong recognition capabilities for dermatologic disease images, can accurately describe abnormalities in images completely, and shows high sensitivity to malignant diseases. Apart from medical assistance, Claude3 Opus could potentially be widely used in medical education and patient communication. CLINICALTRIAL no need

show abstract

The Impact of Artificial Intelligence on Health Equity in Dermatology

Rinderknecht,

Nwandu,

Daneshjou

et al. 2024

Curr Derm Rep

View full text Add to dashboard Cite

Health equity assessment of machine learning performance (HEAL): a framework and dermatology AI model case study

Cited by 4 publications

References 44 publications

Technology innovation to reduce health inequality in skin diagnosis and to improve patient outcomes for people of color: a thematic literature review and future research agenda

Technology innovation to reduce health inequality in skin diagnosis and to improve patient outcomes for people of color: a thematic literature review and future research agenda

Exploring the analysis capabilities and clinical application potential of the Claude 3 Opus in different dermatologic images: the development of a large-scale multimodal model to assist in dermatology clinical practice (Preprint)

The Impact of Artificial Intelligence on Health Equity in Dermatology

Contact Info

Product

Resources

About