Systematic Review of Artificial Intelligence for Abnormality Detection in High-volume Neuroimaging and Subgroup Meta-analysis for Intracranial Hemorrhage Detection

Abstract

Purpose Most studies evaluating artificial intelligence (AI) models that detect abnormalities in neuroimaging are either tested on unrepresentative patient cohorts or are insufficiently well-validated, leading to poor generalisability to real-world tasks. The aim was to determine the diagnostic test accuracy and summarise the evidence supporting the use of AI models performing first-line, high-volume neuroimaging tasks. Methods Medline, Embase, Cochrane library and Web of Science were searched until September 2021 for studies that temporally or externally validated AI capable of detecting abnormalities in first-line computed tomography (CT) or magnetic resonance (MR) neuroimaging. A bivariate random effects model was used for meta-analysis where appropriate. This study was registered on PROSPERO as CRD42021269563. Results Out of 42,870 records screened, and 5734 potentially eligible full texts, only 16 studies were eligible for inclusion. Included studies were not compromised by unrepresentative datasets or inadequate validation methodology. Direct comparison with radiologists was available in 4/16 studies and 15/16 had a high risk of bias. Meta-analysis was only suitable for intracranial hemorrhage detection in CT imaging (10/16 studies), where AI systems had a pooled sensitivity and specificity 0.90 (95% confidence interval [CI] 0.85–0.94) and 0.90 (95% CI 0.83–0.95), respectively. Other AI studies using CT and MRI detected target conditions other than hemorrhage (2/16), or multiple target conditions (4/16). Only 3/16 studies implemented AI in clinical pathways, either for pre-read triage or as post-read discrepancy identifiers. Conclusion The paucity of eligible studies reflects that most abnormality detection AI studies were not adequately validated in representative clinical cohorts. The few studies describing how abnormality detection AI could impact patients and clinicians did not explore the full ramifications of clinical implementation.

Optimising brain age estimation through transfer learning: A suite of pre‐trained foundation models for improved performance and generalisability in a clinical setting

“…This step successfully overcame a significant obstacle often faced in brain age model development (i.e. identifying radiological normal scans in a large hospital dataset), resulting in a diverse and realistic set of training data that accurately represents clinical populations (Agarwal et al, 2023; Booth et al, 2023; Din et al, 2023). The diversity of our data, encompassing a range of scanner vendors, acquisition protocols, patient ethnicities, and a wide age span (18–96 years), added robustness to our models.…”

“…However, realising this goal will involve overcoming several challenges. One challenge is the lack of representativeness of research datasets (Agarwal et al, 2023; Agarwal & Wood et al, 2023; Din et al, 2023), particularly public datasets commonly used for training brain age models. This applies not only to the demographics of the study participants, but also to the nature of the MRI data (e.g.…”

Factors affecting the labelling accuracy of brain MRI studies relevant for deep learning abnormality detection

“…Specifically, in a theoretical “triage-based” Artificial Intelligence (AI) clinical pathway, a false-negative AI report (engendered by a low recall score for the model) may place the incorrectly-reported study to the bottom of the queue for human reporting and potentially significantly delay diagnosis. The exact definition of a “good enough” F1-score for neuroimaging applications remains to be determined and indeed, at the time of writing, almost no neuroimaging abnormality detection tools have been adequately validated in representative clinical cohorts ( 10 ), although there are a few notable exceptions ( 7 , 8 ).…”

The novel imaging methods in diagnosis and assessment of cerebrovascular diseases: an overview

“…They can aid radiologists in the quantification and characterization of lesions, providing more accurate and reproducible measurements ( 133 ). Additionally, AI algorithms can help predict patient outcomes, such as the risk of recurrent strokes or response to treatment, based on imaging findings and clinical data ( 134 , 135 ). While, a study conducted by Voter AF and colleagues showed unexpectedly lower sensitivity and positive predictive values for Aidoc in diagnosing intracranial hemorrhage, which has raised concerns about the generalizability of these commercial AI tools ( 136 ).…”