Generalizability and usefulness of artificial intelligence for skin cancer diagnostics: An algorithm validation study

Abstract

Background: Artificial intelligence can be trained to outperform dermatologists in image-based skin cancer diagnostics. However, the networks' sensitivity to biases and overfitting may hamper their clinical applicability. Objectives:The aim of this study was to explain the potential consequences of implementing convolutional neural networks for stand-alone melanoma diagnostics and skin lesion triage. Methods:In this algorithm validation study on retrospective data, we reproduced and evaluated the performance of state-of-the-art artificial intelligence (convolutional neural networks) for skin cancer diagnostics. The networks were trained on 25,331 annotated dermoscopic skin lesion images from an open-source data set (ISIC-2019) and tested using a novel data set (AISC-2021) consisting of 26,591 annotated dermoscopic skin lesion images. We tested the trained algorithms' ability to generalize to new data and their diagnostic performance in two simulations (melanoma diagnostics and skin lesion triage). Results: The trained algorithms performed significantly less accurate diagnostics on images of nevi, melanomas and actinic keratoses from the AISC-2021 data set than the ISIC-2019 data set (p < 0.003). Almost one-third (31.1%) of the melanomas were misclassified during the melanoma diagnostics simulation, irrespective of their Breslow thickness. Furthermore, the algorithms marked 92.7% of the lesions 'suspicious' during the triage simulation, which yielded a triage sensitivity and specificity of 99.7% and 8.2%, respectively. Conclusions: Although state-of-the-art artificial intelligence outperforms dermatologists on image-based skin lesion classification within an artificial

Position statement of the EADV Artificial Intelligence (AI) Task Force on AI‐assisted smartphone apps and web‐based services for skin disease

“…To ensure adequate training, image data should include skin conditions relevant to the target population, along with patient-level contextual information. 7,8 Especially for apps and web-based services that are primarily intended to be used by consumers, it is expected that a large variety of lesion types will be photographed for signs of skin cancer (e.g. inflammatory lesions, scars, angioma, benign nevi).…”

“…10,11 The training data should include images captured using a variety of hardware (cameras) and software (image capturing applications) from the intended setting where the final algorithms will be used. 7,12 To avoid adverse events from algorithm bias in applications intended towards consumers, the training library should include a large number of non-professional photographs taken by consumers. This information can help to ensure that an AI algorithm can be adequately trained based on the relevant skin conditions in the target population and a variety of image sources.…”

“…One of the factors that may contribute to this poor accuracy is the quality, variance and quantity of image training data used to develop and validate AI algorithms. To ensure adequate training, image data should include skin conditions relevant to the target population, along with patient‐level contextual information 7,8 . Especially for apps and web‐based services that are primarily intended to be used by consumers, it is expected that a large variety of lesion types will be photographed for signs of skin cancer (e.g.…”

“…early Merkel cell carcinomas) to avoid potential biases that could result in inaccurate outcomes 10,11 . The training data should include images captured using a variety of hardware (cameras) and software (image capturing applications) from the intended setting where the final algorithms will be used 7,12 . To avoid adverse events from algorithm bias in applications intended towards consumers, the training library should include a large number of non‐professional photographs taken by consumers.…”

Validity evidence supporting clinical skills assessment by artificial intelligence compared with trained clinician raters

“…This has implications for the interpretation of our study results as we may understate the potential value of AI‐based assessments. The robustness of the AIBA model may be improved by including more diverse training sets and by adjusting the model's hyperparameters to avoid overfitting to the training dataset 46–49 . However, the need for cross‐validation and very large datasets may ultimately hinder the accessibility and use of AI for assessment purposes, in particular, when compared with EBA that work after minimal rater instruction.…”