Effect of a comprehensive deep-learning model on the accuracy of chest x-ray interpretation by radiologists: a retrospective, multireader multicase study

Abstract

Background Chest x-rays are widely used in clinical practice; however, interpretation can be hindered by human error and a lack of experienced thoracic radiologists. Deep learning has the potential to improve the accuracy of chest x-ray interpretation. We therefore aimed to assess the accuracy of radiologists with and without the assistance of a deeplearning model. MethodsIn this retrospective study, a deep-learning model was trained on 821 681 images (284 649 patients) from five data sets from Australia, Europe, and the USA. 2568 enriched chest x-ray cases from adult patients (≥16 years) who had at least one frontal chest x-ray were included in the test dataset; cases were representative of inpatient, outpatient, and emergency settings. 20 radiologists reviewed cases with and without the assistance of the deep-learning model with a 3-month washout period. We assessed the change in accuracy of chest x-ray interpretation across 127 clinical findings when the deep-learning model was used as a decision support by calculating area under the receiver operating characteristic curve (AUC) for each radiologist with and without the deep-learning model. We also compared AUCs for the model alone with those of unassisted radiologists. If the lower bound of the adjusted 95% CI of the difference in AUC between the model and the unassisted radiologists was more than -0•05, the model was considered to be non-inferior for that finding. If the lower bound exceeded 0, the model was considered to be superior. Findings Unassisted radiologists had a macroaveraged AUC of 0•713 (95% CI 0•645-0•785) across the 127 clinical findings, compared with 0•808 (0•763-0•839) when assisted by the model. The deep-learning model statistically significantly improved the classification accuracy of radiologists for 102 (80%) of 127 clinical findings, was statistically non-inferior for 19 (15%) findings, and no findings showed a decrease in accuracy when radiologists used the deeplearning model. Unassisted radiologists had a macroaveraged mean AUC of 0•713 (0•645-0•785) across all findings, compared with 0•957 (0•954-0•959) for the model alone. Model classification alone was significantly more accurate than unassisted radiologists for 117 (94%) of 124 clinical findings predicted by the model and was non-inferior to unassisted radiologists for all other clinical findings. Interpretation This study shows the potential of a comprehensive deep-learning model to improve chest x-ray interpretation across a large breadth of clinical practice. Funding Annalise.ai.

An Artificial Intelligence–Based Chest X-ray Model on Human Nodule Detection Accuracy From a Multicenter Study

“…Although our study did not assess the clinical relevance of pulmonary nodules with AI-aided interpretation as opposed to unaided interpretation, the importance of our findings can be inferred from a 2021 targeted study on improved detection of malignant pulmonary nodules with AI algorithms. 17 Another implication of our study is the improved sensitivity of junior radiologists with AI-aided interpretation, because such improvement can help radiologists detect (or finesse their search patterns for) pulmonary nodules on chest radiographs. Greater improvement in specificity for senior radiologists vs their junior colleagues implies that AI outputs can increase radiologists' confidence for ruling out pulmonary nodules and to avoid labeling non-nodular abnormalities and normal structures as pulmonary nodules.…”

“…Although our study did not assess the clinical relevance of pulmonary nodules with AI-aided interpretation as opposed to unaided interpretation, the importance of our findings can be inferred from a 2021 targeted study on improved detection of malignant pulmonary nodules with AI algorithms. 17 …”

Assessment of the effect of a comprehensive chest radiograph deep learning model on radiologist reports and patient outcomes: a real-world observational study

“…A modified version of a commercially available AI tool for use as a diagnostic assist device displaying results within a viewer (CXR viewer; Annalise CXR V.1.2, Annalise-AI, Sydney, Australia) was evaluated. 32 The AI tool deploys an underlying machine learning model, developed and validated by Seah et al , 31 which consists of attribute and classification CNNs based on the EfficientNet architecture 33 and a segmentation CNN based on U-Net 34 with EfficientNet backbone. The model was trained on 821 681 de-identified CXR images from 284 649 patients originating from inpatient, outpatient and emergency settings across Australia, Europe and North America.…”

“…The model was validated for 124 clinical findings in a multireader, multicase study. 31 Thirty-four of these findings were deemed priority findings based on their clinical importance. The full list of 124 findings is available in online supplemental table 1 .…”

“…Recently, our group developed a comprehensive deep learning CXR diagnostic assist device, which was designed to assist clinicians in CXR interpretation and improve diagnostic accuracy, validated for 124 clinically relevant findings seen on frontal and lateral chest radiographs. 31 The primary objective of the current study was to evaluate the real-world usefulness of the model as a diagnostic assist device for radiologists in both hospital and community clinic settings. This involved examining the frequency at which the model’s recommendations led to a ‘significant impact on the report’, defined as the inclusion of findings recommended by the model which altered the radiologists report in a meaningful way.…”