Performance of a Deep Learning Algorithm Compared with Radiologic Interpretation for Lung Cancer Detection on Chest Radiographs in a Health Screening Population

Abstract

Background: The performance of a deep learning algorithm for lung cancer detection on chest radiographs in a health screening population is unknown. Purpose: To validate a commercially available deep learning algorithm for lung cancer detection on chest radiographs in a health screening population. Materials and Methods: Out-of-sample testing of a deep learning algorithm was retrospectively performed using chest radiographs from individuals undergoing a comprehensive medical checkup between July 2008 and December 2008 (validation test). To evaluate the algorithm performance for visible lung cancer detection, the area under the receiver operating characteristic curve (AUC) and diagnostic measures, including sensitivity and false-positive rate (FPR), were calculated. The algorithm performance was compared with that of radiologists using the McNemar test and the Moskowitz method. Additionally, the deep learning algorithm was applied to a screening cohort undergoing chest radiography between January 2008 and December 2012, and its performances were calculated. Results: In a validation test comprising 10 285 radiographs from 10 202 individuals (mean age, 54 years 6 11 [standard deviation]; 5857 men) with 10 radiographs of visible lung cancers, the algorithm's AUC was 0.99 (95% confidence interval: 0.97, 1), and it showed comparable sensitivity (90% [nine of 10 radiographs]) to that of the radiologists (60% [six of 10 radiographs]; P = .25) with a higher FPR (3.1% [319 of 10 275 radiographs] vs 0.3% [26 of 10 275 radiographs]; P , .001). In the screening cohort of 100 525 chest radiographs from 50 070 individuals (mean age, 53 years 6 11; 28 090 men) with 47 radiographs of visible lung cancers, the algorithm's AUC was 0.97 (95% confidence interval: 0.95, 0.99), and its sensitivity and FPR were 83% (39 of 47 radiographs) and 3% (2999 of 100 478 radiographs), respectively. Conclusion: A deep learning algorithm detected lung cancers on chest radiographs with a performance comparable to that of radiologists, which will be helpful for radiologists in healthy populations with a low prevalence of lung cancer.

Use of Artificial Intelligence-Based Software as Medical Devices for Chest Radiography: A Position Paper from the Korean Society of Thoracic Radiology

“…However, the reproducibility of those kjronline.org performances in the actual practice remains unclear, since retrospectively collected data may not fully reflect the prevalence and diversity of abnormalities in the actual clinical situation [10,11]. Several recent investigations reported excellent performance of AI-SaMDs for the identification of specific abnormalities or diseases such as pulmonary nodules [12], tuberculosis [13][14][15], and coronavirus disease pneumonia [16,17], in consecutive cohorts reflecting actual clinical situations. Nevertheless, further investigations validating the performance of AI-SaMDs during their utilization in the real clinical practice should be conducted to confirm the reproducibility of such in the daily practice.…”

Artificial Intelligence-Based Detection of Pneumonia in Chest Radiographs

“…It comprises ten radio-morphological patterns including pulmonal nodules, pneumothorax, fibrosis, atelectasis, cardiomegaly, calcification, pleural effusion, pneumoperitoneum, mediastinal widening and consolidations/opacifications. Other studies have proved that this program’s performance in detecting the aforementioned imaging patterns is excellent [ 11 , 31 , 32 ]. With the recommended cut-off value of 15 for the consolidation score, a high sensitivity (95.4%) and moderate specificity (66.0%) were achieved, which is optimal for application in the context of an automated preliminary assessment.…”

Deep learning applied to automatic disease detection using chest X‐rays

“…Cha 28 developed a DCNN model for detecting operable lung cancer demonstrating sensitivity superior to human readers. Lee 29 assessed the performance of a commercially available DL algorithm when compared with human interpretation for lung cancer detection on CXR in a health screening population. The AUROC was 0.99 with sensitivity 90% and FPR 3.1% compared to the average sensitivity of the radiologists which was 60% with FPR of 0.3%.…”