Accuracy of artificial intelligence software for CT angiography in stroke

Abstract

Objective Software developed using artificial intelligence may automatically identify arterial occlusion and provide collateral vessel scoring on CT angiography (CTA) performed acutely for ischemic stroke. We aimed to assess the diagnostic accuracy of e‐CTA by Brainomix™ Ltd by large‐scale independent testing using expert reading as the reference standard. Methods We identified a large clinically representative sample of baseline CTA from 6 studies that recruited patients with acute stroke symptoms involving any arterial territory. We compared e‐CTA results with masked expert interpretation of the same scans for the presence and location of laterality‐matched arterial occlusion and/or abnormal collateral score combined into a single measure of arterial abnormality. We tested the diagnostic accuracy of e‐CTA for identifying any arterial abnormality (and in a sensitivity analysis compliant with the manufacturer's guidance that software only be used to assess the anterior circulation). Results We include CTA from 668 patients (50% female; median: age 71 years, NIHSS 9, 2.3 h from stroke onset). Experts identified arterial occlusion in 365 patients (55%); most (343, 94%) involved the anterior circulation. Software successfully processed 545/668 (82%) CTAs. The sensitivity, specificity and diagnostic accuracy of e‐CTA for detecting arterial abnormality were each 72% (95% CI = 66–77%). Diagnostic accuracy was non‐significantly improved in a sensitivity analysis excluding occlusions from outside the anterior circulation (76%, 95% CI = 72–80%). Interpretation Compared to experts, the diagnostic accuracy of e‐CTA for identifying acute arterial abnormality was 72–76%. Users of e‐CTA should be competent in CTA interpretation to ensure all potential thrombectomy candidates are identified.

Real-world evaluation of Brainomix e-Stroke software

“…Both of these values are higher than a prior study based on the retrospective analysis of 545 CTAs from trial data (sensitivity and specificity of 72%). 21 Again, some of this difference may be related to the incidence of large vessel occlusions (53.5% vs 12.5% in this study) and different software versions. Medium vessel occlusions are outside the specifications of the software and therefore, as would be expected, performance for medium vessel occlusions is significantly lower than for large vessel occlusions.…”

“…Identifying hyperdense vessels due to acute thrombus was the least accurate (69.1%) component of e-ASPECTS, which was hampered by a large number of false positives e-ASPECTS was highly accurate in detecting acute haemorrhage, which reflects prior studies. 21 Accuracy was limited only by false positives mainly caused by choroid plexus calcification. While high sensitivity to acute haemorrhage is important prior to initiating timesensitive thrombolysis, false positives carry the risk of causing an unnecessary delay while a specialist review is sought.…”