Localization-adjusted diagnostic performance and assistance effect of a computer-aided detection system for pneumothorax and consolidation

Abstract

While many deep-learning-based computer-aided detection systems (CAD) have been developed and commercialized for abnormality detection in chest radiographs (CXR), their ability to localize a target abnormality is rarely reported. Localization accuracy is important in terms of model interpretability, which is crucial in clinical settings. Moreover, diagnostic performances are likely to vary depending on thresholds which define an accurate localization. In a multi-center, stand-alone clinical trial using temporal and external validation datasets of 1,050 CXRs, we evaluated localization accuracy, localization-adjusted discrimination, and calibration of a commercially available deep-learning-based CAD for detecting consolidation and pneumothorax. The CAD achieved image-level AUROC (95% CI) of 0.960 (0.945, 0.975), sensitivity of 0.933 (0.899, 0.959), specificity of 0.948 (0.930, 0.963), dice of 0.691 (0.664, 0.718), moderate calibration for consolidation, and image-level AUROC of 0.978 (0.965, 0.991), sensitivity of 0.956 (0.923, 0.978), specificity of 0.996 (0.989, 0.999), dice of 0.798 (0.770, 0.826), moderate calibration for pneumothorax. Diagnostic performances varied substantially when localization accuracy was accounted for but remained high at the minimum threshold of clinical relevance. In a separate trial for diagnostic impact using 461 CXRs, the causal effect of the CAD assistance on clinicians’ diagnostic performances was estimated. After adjusting for age, sex, dataset, and abnormality type, the CAD improved clinicians’ diagnostic performances on average (OR [95% CI] = 1.73 [1.30, 2.32]; p < 0.001), although the effects varied substantially by clinical backgrounds. The CAD was found to have high stand-alone diagnostic performances and may beneficially impact clinicians’ diagnostic performances when used in clinical settings.

Deep learning for pneumothorax diagnosis: a systematic review and meta-analysis

“…Among the 63 studies, 56 studies identified pneumothorax on chest radiography [ 26 – 81 ], four studies on computed tomography [ 82 – 85 ], one study on ECG [ 86 ], one study used chest radiography and photography using a smartphone [ 87 ], and one study used chest radiography and tabular data [ 88 ]. Six studies developed and internally tuned DLs [ 37 , 52 , 63 , 67 , 74 , 76 ], 25 studies also internally tested their DLs [ 32 , 33 , 35 , 38 , 40 , 41 , 43 , 45 , 47 , 48 , 50 , 55 , 60 , 65 , 69 , 70 , 73 , 75 , 79 – 83 , 85 , 86 ] and 32 studies externally tested the DLs [ 26 – 31 , 34 , 36 , 39 , 42 , 44 , 46 , 49 , 51 , 53 , 54 , 56 – 59 , 61 , 62 , 64 , 66 , 68 , 71 , 72 , 77 , 78 , 84 , 87 , 88 ].…”

“…Five studies compared the performance of DL with physicians: two studies compared DL with experts (not a resident or technologist) [ 42 , 83 ], two studies compared DL with both experts and non-experts [ 43 , 68 ] and one study compared DL with non-experts only [ 34 ]. Two studies compared the performance of DL with radiology reports written in daily clinical practice [ 42 , 56 ].…”

“…Two studies compared the performance of DL with radiology reports written in daily clinical practice [ 42 , 56 ]. Two studies included physician performance with and without DL assistance as a comparison group [ 44 , 68 ]. Detailed physician characteristics are shown in supplementary table S4 .…”

“…As for model development, to generate a reference standard for image labelling, 18 studies used expert consensus [ 27 – 33 , 35 – 38 , 49 , 53 – 55 , 71 , 77 , 83 ], two relied on the opinion of a single expert reader [ 76 , 85 ], 16 used pre-existing radiological reports or other imaging modalities [ 34 , 41 , 43 , 45 , 46 , 52 , 60 , 61 , 67 , 75 , 78 – 82 , 87 ], one study defined their reference standard as surgical confirmation (indicated for surgery) [ 86 ], 11 studies used mixed methods (any combination of the aforementioned) [ 40 , 47 , 48 , 50 , 51 , 62 , 63 , 65 , 69 , 70 , 73 ] and two studies did not report how their reference standard was generated [ 74 , 88 ]. As for model testing, to generate a reference standard for image labelling, 26 studies used expert consensus [ 26 – 28 , 30 – 33 , 38 , 39 , 44 , 51 , 54 – 57 , 61 , 64 , 66 , 68 , 71 – 73 , 77 , 80 , 83 , 84 ], two relied on the opinion of a single expert reader [ 58 , 85 ], 11 used pre-exist...…”

“…In studies that performed external model validation, the median dataset size was 1137 (range 175–112 120). 17 studies included localisation of pneumothorax in model output to improve end-user interpretability [ 26 – 28 , 30 – 33 , 36 – 38 , 40 , 47 , 56 , 59 , 68 , 84 , 85 ]. Detailed DL characteristics are shown in supplementary table S3 .…”