Tuberculosis detection from chest x-rays for triaging in a high tuberculosis-burden setting: an evaluation of five artificial intelligence algorithms

Abstract

Background Artificial intelligence (AI) algorithms can be trained to recognise tuberculosis-related abnormalities on chest radiographs. Various AI algorithms are available commercially, yet there is little impartial evidence on how their performance compares with each other and with radiologists. We aimed to evaluate five commercial AI algorithms for triaging tuberculosis using a large dataset that had not previously been used to train any AI algorithms.Methods Individuals aged 15 years or older presenting or referred to three tuberculosis screening centres in Dhaka, Bangladesh, between May 15, 2014, and Oct 4, 2016, were recruited consecutively. Every participant was verbally screened for symptoms and received a digital posterior-anterior chest x-ray and an Xpert MTB/RIF (Xpert) test. All chest x-rays were read independently by a group of three registered radiologists and five commercial AI algorithms: CAD4TB (version 7), InferRead DR (version 2), Lunit INSIGHT CXR (version 4.9.0), JF CXR-1 (version 2), and qXR (version 3). We compared the performance of the AI algorithms with each other, with the radiologists, and with the WHO's Target Product Profile (TPP) of triage tests (≥90% sensitivity and ≥70% specificity). We used a new evaluation framework that simultaneously evaluates sensitivity, proportion of Xpert tests avoided, and number needed to test to inform implementers' choice of software and selection of threshold abnormality scores. Findings Chest x-rays from 23 954 individuals were included in the analysis. All five AI algorithms significantly outperformed the radiologists. The areas under the receiver operating characteristic curve were 90•81% (95% CI 90•33-91•29) for qXR, 90•34% (89•81-90•87) for CAD4TB, 88•61% (88•03-89•20) for Lunit INSIGHT CXR, 84•90% (84•27-85•54) for InferRead DR, and 84•89% (84•26-85•53) for JF CXR-1. Only qXR (74•3% specificity [95% CI 73•3-74•9]) and CAD4TB (72•9% specificity [72•3-73•5]) met the TPP at 90% sensitivity. All five AI algorithms reduced the number of Xpert tests required by 50% while maintaining a sensitivity above 90%. All AI algorithms performed worse among older age groups (>60 years) and people with a history of tuberculosis.Interpretation AI algorithms can be highly accurate and useful triage tools for tuberculosis detection in high-burden regions, and outperform human readers.Funding Government of Canada.

Independent evaluation of 12 artificial intelligence solutions for the detection of tuberculosis

“…This is also the first time six of these CAD solutions have been independently evaluated in the literature. Previous systematic reviews have focused solely on Delft Imaging’s CAD4TB 24 , 25 , and more recent comparative evaluations 26 , 27 , 29 have included only a limited number of CAD solutions. This independent evaluation highlights the recent significant advances in diagnostic accuracy of multiple CAD software platforms and also identifies important limitations of the CAD software, which should be addressed in future implementation research.…”

“…A strength of this CAD software evaluation was the involvement of two TB clinicians with different levels of experience as benchmarks for the software solutions. This particularly pertains to the inclusion of the Intermediate Reader, as many CAD software evaluations have used a single highly skilled radiologist to re-read the CXR images, thereby setting a very high standard for CAD software diagnostic accuracy 23 , 27 , 29 . However, experienced expert TB clinicians and radiologists are unlikely to participate in programmatic CXR screening initiatives on a regular basis.…”

“…It is possible that the CAD software developers who received DICOM files from FIT had their own radiologists rapidly grade the test library so they could use their radiologist’s interpretations to influence or adjust their CAD software outputs before providing them to FIT. However, this likelihood was deemed to be low, particularly for commercially available CAD solutions, and recent CAD software evaluations have used similar methods for data collection 27 , 29 . To highlight the differences in data collection methods, and higher levels of trust in the CAD software outputs directly collected by FIT, all analyses in this manuscript have been presented by data collection method.…”

“…Two systematic reviews, conducted in 2016 and 2019, also primarily included studies evaluating various versions of the CAD4TB software 24 , 25 . More recent evaluations have included additional CAD software solutions, including Qure.ai’s qXR (India), Lunit’s INSIGHT CXR (South Korea), JF Healthcare’s JF CXR-1 (China) and InferVision’s InferRead DR Chest (Japan) 26 – 29 . These early evaluations suggest that CAD solutions can match the performance of experienced human readers for detecting abnormalities associated with TB.…”

Clinical and radiographic characteristics of presumptive tuberculosis patients previously treated for tuberculosis in Zambia

“…The PPV of chest radiography for active TB among presumptive patients with a history of TB treatment in this study (21%) was lower than was previously found in Lusaka, Zambia among presumptive TB patients regardless of TB treatment history (33%) using chest radiography with computer-aided detection software [ 21 ]. In general, the performance of chest radiography as a screening and triage tool for TB disease according to prior TB treatment status is poorly characterized; however, a recent study from Bangladesh demonstrated that chest radiography using several different computer-aided detection software had substantially reduced diagnostic performance for active TB among those previously treated for TB [ 22 ]. Thus, it is important that future studies evaluate the accuracy of chest radiography and other TB screening and triage tools suitable for use in resource-limited settings (e.g., screening algorithms, prediction models, point-of-care tests), according to prior TB treatment status, to define optimal diagnostic strategies for this important patient population.…”