Over-Detection of Melanoma-Suspect Lesions by a CE-Certified Smartphone App: Performance in Comparison to Dermatologists, 2D and 3D Convolutional Neural Networks in a Prospective Data Set of 1204 Pigmented Skin Lesions Involving Patients’ Perception

Abstract

The exponential increase in algorithm-based mobile health (mHealth) applications (apps) for melanoma screening is a reaction to a growing market. However, the performance of available apps remains to be investigated. In this prospective study, we investigated the diagnostic accuracy of a class 1 CE-certified smartphone app in melanoma risk stratification and its patient and dermatologist satisfaction. Pigmented skin lesions ≥3 mm and any suspicious smaller lesions were assessed by the smartphone app SkinVision® (SkinVision® B.V., Amsterdam, the Netherlands, App-Version 6.8.1), 2D FotoFinder ATBM® master (FotoFinder ATBM® Systems GmbH, Bad Birnbach, Germany, Version 3.3.1.0), 3D Vectra® WB360 (Canfield Scientific, Parsippany, New Jersey, USA, Version 4.7.1) total body photography (TBP) devices, and dermatologists. The high-risk score of the smartphone app was compared with the two gold standards: histological diagnosis, or if not available, the combination of dermatologists’, 2D and 3D risk assessments. A total of 1204 lesions among 114 patients (mean age 59 years; 51% females (55 patients at high-risk for developing a melanoma, 59 melanoma patients)) were included. The smartphone app’s sensitivity, specificity, and area under the receiver operating characteristics (AUROC) varied between 41.3–83.3%, 60.0–82.9%, and 0.62–0.72 according to two study-defined reference standards. Additionally, all patients and dermatologists completed a newly created questionnaire for preference and trust of screening type. The smartphone app was rated as trustworthy by 36% (20/55) of patients at high-risk for melanoma, 49% (29/59) of melanoma patients, and 8.8% (10/114) of dermatologists. Most of the patients rated the 2D TBP imaging (93% (51/55) resp. 88% (52/59)) and the 3D TBP imaging (91% (50/55) resp. 90% (53/59)) as trustworthy. A skin cancer screening by combination of dermatologist and smartphone app was favored by only 1.8% (1/55) resp. 3.4% (2/59) of the patients; no patient preferred an assessment by a smartphone app alone. The diagnostic accuracy in clinical practice was not as reliable as previously advertised and the satisfaction with smartphone apps for melanoma risk stratification was scarce. MHealth apps might be a potential medium to increase awareness for melanoma screening in the lay population, but healthcare professionals and users should be alerted to the potential harm of over-detection and poor performance. In conclusion, we suggest further robust evidence-based evaluation before including market-approved apps in self-examination for public health benefits.

Position statement of the EADV Artificial Intelligence (AI) Task Force on AI‐assisted smartphone apps and web‐based services for skin disease

“…In fact, recent studies have shown generally poor accuracy of these apps in terms of sensitivity and/or specificity in detecting skin cancer. [4][5][6] Quality, quantity and variance of image training data One of the factors that may contribute to this poor accuracy is the quality, variance and quantity of image training data used to develop and validate AI algorithms. To ensure adequate training, image data should include skin conditions relevant to the target population, along with patient-level contextual information.…”

“…Smartphone apps may cause significant adverse events through inaccurate predictions, which can be falsely reassuring or falsely concerning. In fact, recent studies have shown generally poor accuracy of these apps in terms of sensitivity and/or specificity in detecting skin cancer 4–6 …”

“…While it is challenging to state the minimally acceptable accuracy that these apps should offer, the task force emphasizes the importance of evaluating these apps based on their overall benefits while minimising potential risk and potential burden on the overall healthcare system derived from these apps. 6,[17][18][19] Currently, there is a distinct lack of studies that demonstrate this benefit-risk relationship. We recommend that studies should be performed prospectively, in the intended-use setting, with outcome measures directly related to the benefits of patients.…”

Diagnostics Using Non-Invasive Technologies in Dermatological Oncology

“…Some of these instruments allow a faster and more accurate diagnosis of skin neoplasms, which is necessary to ensure adequate treatment of the patient. At the same time, they require specific, lengthy training and may increase costs to health systems if used inappropriately [ 131 , 132 , 133 ].…”

Recent Progress in the Diagnosis and Treatment of Melanoma and Other Skin Cancers

“…Therefore, smartphone applications based on artificial intelligence (AI) that support skin self-exams are becoming increasingly popular [ 4 ]. Jahn et al, 2022 investigated the diagnostic accuracy of the SkinVision ® application in classifying pigmented lesions as benign or possible melanoma [ 5 ]. The sensitivity and specificity in the differentiation of nevi and melanoma were low, with 41.3–83.3% and 60.0–82.9%, respectively.…”