Performance of a Chest Radiography AI Algorithm for Detection of Missed or Mislabeled Findings: A Multicenter Study

Abstract

We assessed whether a CXR AI algorithm was able to detect missed or mislabeled chest radiograph (CXR) findings in radiology reports. Methods: We queried a multi-institutional radiology reports search database of 13 million reports to identify all CXR reports with addendums from 1999–2021. Of the 3469 CXR reports with an addendum, a thoracic radiologist excluded reports where addenda were created for typographic errors, wrong report template, missing sections, or uninterpreted signoffs. The remaining reports contained addenda (279 patients) with errors related to side-discrepancies or missed findings such as pulmonary nodules, consolidation, pleural effusions, pneumothorax, and rib fractures. All CXRs were processed with an AI algorithm. Descriptive statistics were performed to determine the sensitivity, specificity, and accuracy of the AI in detecting missed or mislabeled findings. Results: The AI had high sensitivity (96%), specificity (100%), and accuracy (96%) for detecting all missed and mislabeled CXR findings. The corresponding finding-specific statistics for the AI were nodules (96%, 100%, 96%), pneumothorax (84%, 100%, 85%), pleural effusion (100%, 17%, 67%), consolidation (98%, 100%, 98%), and rib fractures (87%, 100%, 94%). Conclusion: The CXR AI could accurately detect mislabeled and missed findings. Clinical Relevance: The CXR AI can reduce the frequency of errors in detection and side-labeling of radiographic findings.

Role of an Automated Deep Learning Algorithm for Reliable Screening of Abnormality in Chest Radiographs: A Prospective Multicenter Quality Improvement Study

“…In a conventional system, the normal and abnormal CXR are in the same worklist, and there is no way to segregate/triage normal CXR without opening the CXR. AI as a secondary reader assists in reducing errors in the reports and missed diagnoses [ 22 ]. Beyond reduction in reporting time and improvement in report quality, the use of AI will lead to more appropriate treatments for the patients in a timely manner.…”

“…The commercially available deep learning-based AI algorithm qXR (Qure.ai Technologies, Mumbai, India) [ 20 ] has been used in multiple studies previously in CXR screening for diagnosis of tuberculosis [ 21 ], missed or mislabelled findings [ 22 ], severity assessment of pneumonia with the need for mechanical ventilation [ 23 ], and identification of malignant nodules [ 24 ]. We conducted a prospective multicentre quality-improvement study.…”

Lung cancer screening in Lebanon: Joint statement from the Lebanese Pulmonary Society and the Lebanese Society of Medical Oncology

“…Of note, some population-based cohort studies have underlined the effectiveness of chest radiographies in detecting early lung lesions and in reducing the mortality rate of patients with lung cancer (14)(15)(16). Additionally, in view of the evolution of artificial intelligence in powering readings (17), there is a need for further pilot studies for evaluation, since chest radiographies are low in cost, readily available and are performed in high numbers.…”

Real-world testing of an artificial intelligence algorithm for the analysis of chest X-rays in primary care settings

“…In this context, AI algorithms, including deep and machine learning models can assist in diagnosis, potentially enhancing diagnostic accuracy. However, while AI will not replace professionals, it is essential to acknowledge that the implementation of AI in routine clinical practice must be both safe and effective 20 , 21 . One of the current concerns lies in the fact that many of the studies on applications of new AI models only present in silico validation, a phenomenon called “digital exceptionalism”, without performing external validation in the actual implementation environment.…”