Diagnostic accuracy of chest X-ray interpretation for tuberculosis by three artificial intelligence-based software in a screening use-case: an individual patient meta-analysis of global data

Abstract

Background Chest X-ray (CXR) screening is a useful diagnostic tool to test individuals at high risk of tuberculosis (TB), yet image interpretation requires trained human readers who are in short supply in many high TB burden countries. Therefore, CXR interpretation by computer-aided detection software (CAD) may overcome some of these challenges, but evidence on its accuracy is still limited. We established a CXR library with images and metadata from individuals and risk groups that underwent TB screening in a variety of countries to assess the diagnostic accuracy of three commercial CAD solutions through an individual participant meta-analysis. Methods and findings We collected digital CXRs and demographic and clinical data from 6 source studies involving a total of 2756 participants, 1753 (64%) of whom also had microbiological test information. All CXR images were analyzed with CAD4TB v6 (Delft Imaging), Lunit Insight CXR TB algorithm v4.9.0 (Lunit Inc.), and qXR v2 (Qure.ai) and re-read by an expert radiologist who was blinded to the initial CXR reading, the CAD scores, and participant information. While the performance of CAD varied across source studies, the pooled, meta-analyzed summary receiver operating characteristic (ROC) curves of the three products against a microbiological reference standard were similar, with area under the curves (AUCs) of 76.4 (95% CI 72.1-80.3) for CAD4TB, 83.3 (95% CI 78.4-87.2) for Lunit, and 76.4 (95% CI 72.1-80.3) for qXR. None of the CAD products, or the radiologists, met the targets for a triage test of 90% sensitivity and 70% specificity. At the same sensitivity of the expert radiologist (94.0%), all CAD had slightly lower point estimates for specificity (22.4% (95% CI 16.9-29.0) for CAD4TB, 34.6% (95% CI 25.3-45.1) for qXR, and 41.0% (95% CI 30.1-53.0) for Lunit compared to 45.6% for the expert radiologist). At the same specificity of 45.6%, all CAD products had lower point estimates for sensitivity but overlapping CIs with the sensitivity estimate of the radiologist. Conclusions We showed that, overall, three commercially available CAD products had a reasonable diagnostic accuracy for microbiologically confirmed pulmonary TB and may achieve a sensitivity and specificity that approximates those of experienced radiologists. While threshold setting and cost-effectiveness modelling are needed to inform the optimal implementation of CAD products as part of screening programs, the availability of CAD will assist in scaling up active case finding for TB and hence contribute to TB elimination in these settings.

Diagnostic accuracy of three computer-aided detection systems for detecting pulmonary tuberculosis on chest radiography when used for screening: analysis of an international, multicenter migrants screening study

“…Based on the findings of this study, combined with those of the parallel study conducted by FIND [22], CAD systems may be considered viable as a tool for automated CXR interpretation with regard to TB detection in screening programs, particularly in remote, and/or a high TB burden places where there are limited resources and access to expert radiologists. The use of CAD systems in these areas may even have a wider application and contribute to increase the global TB detection rate.…”

Economic analysis of different throughput scenarios and implementation strategies of computer-aided detection software as a screening and triage test for pulmonary TB

“…We selected four commercially available CE-certified TB-specific CAD software: CAD4TB (Delft Imaging, Netherlands), Lunit INSIGHT CXR (Lunit, South Korea), qXR (Qure.ai, India), and InferRead DR Chest (Infervision, China). Three of these software programs (CAD4TB, qXR, Lunit INSIGHT CXR) were part of WHO's recent evidence review, for which performance evaluations were done by FIND, the International Organization of Migration, Stop TB partnership and McGill University and informed the WHO screening guideline [16,[22][23][24];therefore they were included in this study. In September 2021, two software programs (CAD4TB and InferRead) were included in the Stop TB Partnership's Global Drug Facility (GDF) catalogue, which is why InferRead was added to this study.…”

“…This study was not able to assess the cost effectiveness of the software. According to the literature [23,31], these software programs differ in diagnostic accuracy, which would likely affect the cost effectiveness-something a costing study would not be able to capture. A research study found that the software used to inform the WHO recommendation (qXR, CAD4TB and Lunit) performed significantly better than InferRead, [31].…”

“…cardiomegaly). Also this analysis did not take into account that the accuracy of the different CAD software programs, which may impact the cost-effectiveness [22][23][24]31]. While a comparative analysis of Lunit, qXR and CAD4TB has informed a recent WHO recommendation on surveillance [4] (personal communication C. Denkinger), data on InferRead is limited and suggests a less satisfactory performance [31].…”

“Similar performances but markedly different triaging thresholds in three CAD4TB versions risk systematic errors in TB screening programs”

“…Previous studies showed that triaging thresholds are not universal and require careful adjustment to demographic characteristics, laboratory capacities, budget, healthcare settings and study goals. 2,3,12,13,[4][5][6][7][8][9][10][11] Multiple studies compared the performance among different CAD-software platforms, 11,14,15 but comparisons between software versions are lacking in the literature and are needed to support screening programs when using CAD-software and negotiating software updates.…”