Single Site performance of AI software for stroke detection and Triage

Abstract

Background: Recently developed software utilizing artificial intelligence for fast detection and triage of stroke cases has the potential to accelerate stroke care and improve patient outcomes. We performed this analysis to evaluate the performance and time to notification of one such software RAPID LVO. Methods: We created a consecutive database of 151 patients for whom scans were processed by the software over a period of eight months. The outputs and running times of the software were collected, alongside patient information based on full patient imaging and evaluation. Results: RAPID LVO achieved a low sensitivity of 63.6% and moderate specificity of 85.8%, with average time to notification of 32.5 minutes, limiting the clinical utility of the alerts. Conclusions: RAPID LVO has low sensitivity, moderate specificity and high time-to-notification performance. Our study data demonstrated in particular low overall sensitivity (63%) for distal occlusions (M2-3). The disparity between the observed performance and the performance reported in RAPID LVO FDA clearance demonstrates the importance of independent, multi- center evaluation. The gap between the performance in this study compared to published records of RAPID AI may be due to differences in imaging hardware, software implementation, connectivity or clinical definitions.

Head-to-head comparison of commercial artificial intelligence solutions for detection of large vessel occlusion at a comprehensive stroke center

“…One limitation of this study is that we did not evaluate the impact of these tools on clinical outcomes. Other studies have previously shown a significant improvement in transfer times of LVO patients ( 10 ), but there are mixed results at comprehensive stroke centers ( 9 , 11 – 13 ). This study included a relatively small sample size at a single institution which may limit generalizability.…”

Software with artificial intelligence-derived algorithms for analysing CT brain scans in people with a suspected acute stroke: a systematic review and cost-effectiveness analysis

“…Based on the searches and inclusion criteria described above, a total of 30 publications relating to 22 studies [33][34][35][36][39][40][41][43][44][45][46][48][49][50][51][52]55,56,[59][60][61][62] were included in the review; the results section of this report cites studies using the primary publication and, where this is different, the publication in which the referenced data were reported.…”

“…The studies included in this review evaluated AI-derived software technologies produced by iSchemaView, Viz, Brainomix and Avicenna. For iSchemaView, three studies evaluated Rapid CTA, 33,35,36 two studies evaluated Rapid LVO, 41,55 one study evaluated Rapid CTP 50 and two studies assessed the effects of implementing RapidAI (comprising Rapid CTA and Rapid CTP). 34,49 Eight studies evaluated Viz LVO 40,43,45,46,52,[59][60][61] and one study evaluated Viz ICH.…”

“…However, one study reported that, to assess whether the sample was clinically representative of patients admitted to hospital with stroke, it was prespecified that age, sex, stroke severity, time since symptom onset and final diagnosis of included participants would be similar to data from the UK SSNAp (April 2018 to March 2019; www.strokeaudit.org), pooled randomised controlled trials (RCTs) and registries. 62 A total of 12 of the 22 included studies were conducted in USA, 33,34,39,40,43,45,46,51,52,[59][60][61] 1 study each was conducted in Australia, 36 Canada, 55 Germany 56 and Hungary, 44 3 studies were multicentre studies conducted in USA, Brazil and Switzerland, 41 in USA and the Netherlands, 50 and in the UK and Germany (population validated for applicability to the UK setting using UK SSNAP data); 62 the remaining three studies did not report information on geographical location. 35,48,49 Of the 22 included studies, 8 reported receiving some support from the manufacturers of AI-derived software technologies (including shareholdings, consulting fees and employment in relation to individual study authors), 35,36,41,46,51,56,59,61 3 studies reported receiving no funding, 33,34,52 2 studies were publicly funded 50,62 and 9 studies reported no information about funding.…”

Impact of an automated large vessel occlusion detection tool on clinical workflow and patient outcomes

“…Initial studies have shown that the implementation of an automated LVO detection tool is associated with reductions in transfer times, door-to-treatment times, and hospital and ICU stays, as well as improvements in clinical outcomes ( 18–20 ). Others have found low-to-moderate sensitivity and specificity and slower time to notification of these tools when implemented in the acute setting ( 21 ).…”