“Similar performances but markedly different triaging thresholds in three CAD4TB versions risk systematic errors in TB screening programs”

Abstract

Tuberculosis (TB) screening programs may apply computer-aided diagnostic (CAD) tools on chest radiographs to select people for microbiological sputum examination using a pre-selected triaging threshold. CAD software-updates are frequently introduced and it is unknown whether their use requires adjustment of triaging thresholds. In a community-based screening program in South Africa, we compared the scores between the three recent CAD4TB versions (v5, v6, and v7) and assessed their performance to identify microbiologically-confirmed TB. The performance of all versions was similar (v5: AUC 0.78, v6: AUC 0.79, v7: AUC 0.80; p-values>0.05), but along a 0-100 point scale, each had markedly different score distributions and optimal triaging thresholds (v5: 40, v6: 47, v7: 20). This has the potential to cause confusion within TB screening programs as these tools are increasingly adopted and new versions released. Independent guidance for adapting CAD triaging thresholds for frequently released software updates is needed.

The rise of artificial intelligence reading of chest X-rays for enhanced TB diagnosis and elimination

“…When CAD4TB versions were compared in a community-based screening programme in South Africa, the three latest versions had similar performance; however, the accuracy of triage thresholds varied between versions, further demonstrating that repeated calibration is necessary. 23 Although regulatory agencies, such as the Food & Drug Administration, already have mechanisms to evaluate software updates, the WHO is still establishing its prequalification programme to adapt to the rapid pace of innovation in this field.…”