Effects of a comprehensive brain computed tomography deep-learning model on radiologist detection accuracy: a multireader, multicase study

Abstract

Background: Non-contrast computed tomography of the brain (NCCTB) is commonly used in clinical practice to detect intracranial pathology but is subject to interpretation errors. Machine learning is capable of augmenting clinical decision making and there is an opportunity to apply deep learning to improve the clinical interpretation of NCCTB scans. This retrospective detection accuracy study assessed the performance changes of radiologists assisted by a deep learning model designed to identify many NCCTB clinical findings and also compared the standalone performance of the model with that of unassisted radiologists. Methods: A deep learning model was trained on 212,484 CT scan images of the brain. Thirty-two radiologists each reviewed 2,848 NCCTB cases in a test dataset with and without the assistance of the deep learning model. The consensus of three subspecialist neuroradiologists with access to reports and clinical history was used as a ground truth baseline for comparison. Performance metrics including area under the receiver operating characteristic curve (AUC) were calculated for the unassisted and assisted radiologists. Average assisted and unassisted radiologist performance was also compared to that of the model for each clinical finding. Findings: Use of the deep learning model by radiologists significantly improved interpretation performance. Assisted and unassisted radiologists demonstrated an average AUC of 0.79 and 0.73 across the 22 grouped parent findings and 0.72 and 0.68 across all 189 child findings combined, respectively. When the model was used as an assistant, change in radiologist AUC was positive and significant for 91 child findings and 158 findings were clinically non-inferior. AUC decrements were identified for 17 findings. The model alone demonstrated an average AUC of 0.93 across all 144 model findings. Interpretation: The assistance of a comprehensive NCCTB deep learning model in a non-clinical setting significantly improved radiologist detection accuracy across a wide range of clinical findings. This study demonstrated the potential of the evaluated model to improve NCCTB interpretation.