Assessment of deep neural networks for the diagnosis of benign and malignant skin neoplasms in comparison with dermatologists: A retrospective validation study

Abstract

Background The diagnostic performance of convolutional neural networks (CNNs) for diagnosing several types of skin neoplasms has been demonstrated as comparable with that of dermatologists using clinical photography. However, the generalizability should be demonstrated using a large-scale external dataset that includes most types of skin neoplasms. In this study, the performance of a neural network algorithm was compared with that of dermatologists in both real-world practice and experimental settings. Methods and findings To demonstrate generalizability, the skin cancer detection algorithm (https://rcnn.modelderm.com) developed in our previous study was used without modification. We conducted a retrospective study with all single lesion biopsied cases (43 disorders; 40,331 clinical images from 10,426 cases: 1,222 malignant cases and 9,204 benign cases); mean age (standard deviation [SD], 52.1 [18.3]; 4,701 men [45.1%]) were obtained from the Department of Dermatology, Severance Hospital in Seoul, Korea between January 1, 2008 and March 31, 2019. Using the external validation dataset, the predictions of the algorithm were compared with the clinical diagnoses of 65 attending physicians who had recorded the clinical diagnoses with thorough examinations in real-world practice. In addition, the results obtained by the algorithm for the data of randomly selected batches of 30 patients were compared with those obtained by 44 dermatologists in experimental settings; the dermatologists were only provided with multiple images of each lesion, without clinical information. With regard to the determination of malignancy, the area under the curve (AUC) achieved by the algorithm was 0.863 (95% confidence interval [CI] 0.852–0.875), when unprocessed clinical photographs were used. The sensitivity and specificity of the algorithm at the predefined high-specificity threshold were 62.7% (95% CI 59.9–65.1) and 90.0% (95% CI 89.4–90.6), respectively. Furthermore, the sensitivity and specificity of the first clinical impression of 65 attending physicians were 70.2% and 95.6%, respectively, which were superior to those of the algorithm (McNemar test; p < 0.0001). The positive and negative predictive values of the algorithm were 45.4% (CI 43.7–47.3) and 94.8% (CI 94.4–95.2), respectively, whereas those of the first clinical impression were 68.1% and 96.0%, respectively. In the reader test conducted using images corresponding to batches of 30 patients, the sensitivity and specificity of the algorithm at the predefined threshold were 66.9% (95% CI 57.7–76.0) and 87.4% (95% CI 82.5–92.2), respectively. Furthermore, the sensitivity and specificity derived from the first impression of 44 of the participants were 65.8% (95% CI 55.7–75.9) and 85.7% (95% CI 82.4–88.9), respectively, which are values comparable with those of the algorithm (Wilcoxon signed-rank test; p = 0.607 and 0.097). Limitations of this study include the exclusive use of high-quality clinical photographs taken in hospitals and the lack of ethnic diversity in the study population. Conclusions Our algorithm could diagnose skin tumors with nearly the same accuracy as a dermatologist when the diagnosis was performed solely with photographs. However, as a result of limited data relevancy, the performance was inferior to that of actual medical examination. To achieve more accurate predictive diagnoses, clinical information should be integrated with imaging information.

Skin cancer classification via convolutional neural networks: systematic review of studies involving human experts

“…Therefore, comparative studies that are solely based on single images fall short of the clinical routine. Interestingly enough, in these [4,18,24] and other [7,29] studies in which multiple images were provided to human experts, the participants only attained at most equivalent results in comparison with CNN-based classification. Nevertheless, to enable a fair comparison, future reader studies should not only provide clinicians but also provide CNNs with additional close-up images and patient information (e.g.…”

“…Han et al [29] established a direct comparison between the performance of a CNN-based classifier and the results obtained by dermatologists for the binary classification into malignant and benign lesions, as well as the automated discrimination between MM and 31 other skin diseases (see Supplementary Table 7). The authors were the first to provide a clinical image reader study in a clinical setting by incorporating 65 attending clinicians that recorded their diagnoses during thorough examinations in clinical practice.…”

“…In most of the analysed studies (18 out of 19, see Tables 1e3), including those with additional clinical or image data, the recording of the participants' diagnoses took place through webbased rating applications or online questionnaires, thereby substantially differing from the decision-making process occurring in daily clinical practice. Only one study had its participants record their diagnosis during clinical examination of the patient [29]. Under these conditions, the CNN was significantly outperformed by the participating dermatologists, regardless of the classification task.…”

Characteristics of publicly available skin cancer image datasets: a systematic review

“…Attempts to access data at source revealed eight regulated access datasets (table) and three regulated access atlases. [56][57][58] Of the eight datasets, six required ethical committee or institutional approval, 10,50,[53][54][55] one required a £75 fee and a licencing agreement, 51 and one required a competition invitation. 52 Thumbnails of images from Asan, Hallym, SNU, and Severance datasets were available for download, but access to full-size images required formal approval from local data access or ethics committees, 50 or the originating hospitals.…”

Métodos computacionais para detecção de plasticidade vegetal em >i<Tradescantia minima>/i< causada por agentes tóxicos

“…obtiveram acurácias de 84.9% com classificação binária e 49.5% com classificação multiclasse. [189][190] As matrizes de confusão binária obtidas com os resultados obtidos com custom CNN (Gráfico 4.9), mostraram uma alta taxa de predição com um accuracy de 88. Esses resultados demonstram a sensibilidade de Tradescantia minima às nanopartículas de prata, portanto, essa espécie também poderia ser usada como bioindicador de contaminação por AgNPs.…”