Augmenting the accuracy of trainee doctors in diagnosing skin lesions suspected of skin neoplasms in a real-world setting: A prospective controlled before-and-after study

Abstract

Background Although deep neural networks have shown promising results in the diagnosis of skin cancer, a prospective evaluation in a real-world setting could confirm these results. This study aimed to evaluate whether an algorithm (http://b2019.modelderm.com) improves the accuracy of nondermatologists in diagnosing skin neoplasms. Methods A total of 285 cases (random series) with skin neoplasms suspected of malignancy by either physicians or patients were recruited in two tertiary care centers located in South Korea. An artificial intelligence (AI) group (144 cases, mean [SD] age, 57.0 [17.7] years; 62 [43.1%] men) was diagnosed via routine examination with photographic review and assistance by the algorithm, whereas the control group (141 cases, mean [SD] age, 61.0 [15.3] years; 52 [36.9%] men) was diagnosed only via routine examination with a photographic review. The accuracy of the nondermatologists before and after the interventions was compared. Results Among the AI group, the accuracy of the first impression (Top-1 accuracy; 58.3%) after the assistance of AI was higher than that before the assistance (46.5%, P = .008). The number of differential diagnoses of the participants increased from 1.9 ± 0.5 to 2.2 ± 0.6 after the assistance (P < .001). In the control group, the difference in the Top-1 accuracy between before and after reviewing photographs was not significant (before, 46.1%; after, 51.8%; P = .19), and the number of differential diagnoses did not significantly increase (before, 2.0 ± 0.4; after, 2.1 ± 0.5; P = .57). Conclusions In real-world settings, AI augmented the diagnostic accuracy of trainee doctors. The limitation of this study is that the algorithm was tested only for Asians recruited from a single region. Additional international randomized controlled trials involving various ethnicities are required.

Evaluation of Artificial Intelligence–Assisted Diagnosis of Skin Neoplasms: A Single-Center, Paralleled, Unmasked, Randomized Controlled Trial

“…In a previous pilot study (Kim et al, 2022), the Top-1 accuracy of trainees was 47.9%. If 25% enhancement after the assistance were regarded as significant, the sample size was calculated as 548 (alpha ¼ 0.05 and power ¼ 0.8), and we planned to recruit 600 cases (Rosner, 2011).…”

A systematic review and meta-analysis of artificial intelligence versus clinicians for skin cancer diagnosis

“…The Sp was comparable ( P = 0.07). AUC 0.933 Sn 80.24 ± 3.11% Sp 91.57 ± 2.66% Acc 84.97 ± 2.45% BCC diagnosis -Dermatologists: Sn 45.98% ± 21.21 Sp 96.03% ± 6.52 Acc 65% ± 11.7 -Non-dermatologists: Sn 10.71% ±10.53 Sp 98.47% ±3.19 Acc 47.57% ± 6.32 Agarwala et al 50 Public: Triage tool www.triage.com free online system composed of four CNN models (training) Institutional (test) Training: > 200,000 images, > 500 skin conditions Test: 353 images E R B Y 21 US board-certified dermatologists Triage algorithm Multiclass Accuracy of the dermatologist’s was better than the AI accuracy Acc 63.3%; 95% CI 58.0–68.4%) Acc: 69.1% (95% CI 63.7–74.1) Kim et al 51 Public Pre-trained algorithm Institutional: Department of Dermatology, Asan Medical Center, Seoul National University, Bundang Hospital (Test) Training: 721,749 images, 178 disease classes Test: 285 images E P B N -10 attending physicians (11.4 ± 8.8 years’ experience after board certification) -11 dermatology trainees -7 intern doctors Model Dermatology; https://modelderm.com Multiclass There was no direct comparison between AI and clinicians Top-1 of the algorithm Sn 52.2% Sp 93.4% Acc 53.5% Top-2 of the algorithm Sn 69.6% Sp 78.5% Acc 66.0% Top-3 of the algorithm Sn 78.3% Sp 66.1% Acc 70.8% Top-1 Dermatologist Sn 79.3% Sp 90.2% Acc 61.8% Trainees Sn 65.5% Sp 81.3% Acc 46.5% Top-2 Dermatologist Sn 86.2% Sp 82.1% Acc 69.4% Trainees Sn 93.1% Sp 51.8% Acc 54.2% Top-3 Dermatologist Sn 86.2% Sp 79.5% Acc 71.5% Trainees Sn 93,1% Sp 49.1% Acc 54.9% Top-1/Top-2/Top-3 accuracies after assistance were significantly higher than those before assistance AI augmented the diagnostic accuracy of trainee doctors Ba. et al 41 Institutional: Chinese PLA General Hospital & Medical School Dataset: 29,280 Training/validation: 25,773.…”

Exploring the influence of transformer-based multimodal modeling on clinicians’ diagnosis of skin diseases: A quantitative analysis

“… 33 In 2022, Han et al reported the first randomized, prospective clinical trial in dermatology that evaluated the performance of physicians collaborating with AI. 34 They further verified that AI could enhance the accuracy of nonexpert physicians in real-world settings. 34 , 35 Although these studies showed AI's potential for improving the performance of nonspecialists in diagnosing skin diseases, they only used dermoscopic or macroscopic image datasets to train the unimodal CAD systems.…”

“… 34 They further verified that AI could enhance the accuracy of nonexpert physicians in real-world settings. 34 , 35 Although these studies showed AI's potential for improving the performance of nonspecialists in diagnosing skin diseases, they only used dermoscopic or macroscopic image datasets to train the unimodal CAD systems. However, these CAD systems may not accurately simulate the behavior of dermatologists making diagnoses based on multimodal data.…”

“…The third limitation is that the vast majority of the data we use for training comes from Asian populations, and the predictions of the algorithms are heavily dependent on the features of the training data. 34 Therefore, they may exhibit uncertainty in different settings, so the accuracy of our deep learning algorithms and their usefulness in assisting clinicians cannot be generalized to non-Asian populations.…”