Assessing the Accuracy of a Deep Learning Method to Risk Stratify Indeterminate Pulmonary Nodules

Abstract

Rationale: The management of indeterminate pulmonary nodules (IPNs) remains challenging, resulting in invasive procedures and delays in diagnosis and treatment. Strategies to decrease the rate of unnecessary invasive procedures and optimize surveillance regimens are needed. Objectives: To develop and validate a deep learning method to improve the management of IPNs. Methods: A Lung Cancer Prediction Convolutional Neural Network model was trained using computed tomography images of IPNs from the National Lung Screening Trial, internally validated, and externally tested on cohorts from two academic institutions. Measurements and Main Results: The areas under the receiver operating characteristic curve in the external validation cohorts were 83.5% (95% confidence interval [CI], 75.4–90.7%) and 91.9% (95% CI, 88.7–94.7%), compared with 78.1% (95% CI, 68.7–86.4%) and 81.9 (95% CI, 76.1–87.1%), respectively, for a commonly used clinical risk model for incidental nodules. Using 5% and 65% malignancy thresholds defining low- and high-risk categories, the overall net reclassifications in the validation cohorts for cancers and benign nodules compared with the Mayo model were 0.34 (Vanderbilt) and 0.30 (Oxford) as a rule-in test, and 0.33 (Vanderbilt) and 0.58 (Oxford) as a rule-out test. Compared with traditional risk prediction models, the Lung Cancer Prediction Convolutional Neural Network was associated with improved accuracy in predicting the likelihood of disease at each threshold of management and in our external validation cohorts. Conclusions: This study demonstrates that this deep learning algorithm can correctly reclassify IPNs into low- or high-risk categories in more than a third of cancers and benign nodules when compared with conventional risk models, potentially reducing the number of unnecessary invasive procedures and delays in diagnosis.

Artificial intelligence in pulmonary medicine: computer vision, predictive model and COVID-19

“…Most studies did not state whether portions of data were missing or how missing data were handled. Lastly, some studies [39][40][41][42][43][44][45] did not externally validate findings with external datasets, missing a crucial step to evaluate the model's performance with completely independent datasets.…”

Management of screen-detected lung nodules: A Canadian partnership against cancer guidance document

“…This approach has shown promising results to discriminate benign from malignant lung nodules. 64,102,[146][147][148][149] However, these models have a tendency toward overfitting and will require extensive real-world validation to ensure their transportability to other settings. Furthermore, these "black box" algorithms lack obvious clinical interpretations that may limit widespread clinical adoption, though advancements in understanding and interpreting deep networks may help in this regard.…”

“…150,151 Deep neural networks have produced impressive accuracy on traditional image classifications and object detection tasks. [147][148][149][150]152 To demonstrate that medical images can be automatically interpreted effectively to determine if a clinical action is indicated without requiring a human read of the images, the Google team recently showed that screening LDCT can be used to build an end-to-end approach to simulate the radiologist's workflow by performing both localization and lung cancer risk categorization tasks using the whole CT volume. 102 The deep learning algorithm equaled or out-performed radiologist readings.…”

Radiomics-based Management of Indeterminate Lung Nodules? Are We There Yet?

“…In this issue of the Journal , Massion and colleagues (pp. 241–249 ) report the development and external validation of a novel, computer-aided, deep learning–based radiomic model, the Lung Cancer Prediction Convolutional Neural Network (LCP-CNN), to distinguish benign nodules from malignant screen-detected and incidentally detected indeterminate pulmonary nodules ( 8 ).…”

“…Massion and colleagues’ ( 8 ) LCP-CNN model represents another promising radiomic model for the classification of both screen-discovered and incidentally discovered pulmonary nodules. The reported AUCs are excellent at 0.92, 0.84, and 0.92 in the NLST (training set, screen-detected), Vanderbilt University, and Oxford University (validation sets, incidental) sets, respectively ( 8 ). The LCP-CNN model outperformed the clinical Mayo Clinic lung-nodule-malignancy probability model for both external validation sets ( 8 ).…”