Evaluation of Risk of Bias in Neuroimaging-Based Artificial Intelligence Models for Psychiatric Diagnosis

Abstract

ImportanceNeuroimaging-based artificial intelligence (AI) diagnostic models have proliferated in psychiatry. However, their clinical applicability and reporting quality (ie, feasibility) for clinical practice have not been systematically evaluated.ObjectiveTo systematically assess the risk of bias (ROB) and reporting quality of neuroimaging-based AI models for psychiatric diagnosis.Evidence ReviewPubMed was searched for peer-reviewed, full-length articles published between January 1, 1990, and March 16, 2022. Studies aimed at developing or validating neuroimaging-based AI models for clinical diagnosis of psychiatric disorders were included. Reference lists were further searched for suitable original studies. Data extraction followed the CHARMS (Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modeling Studies) and PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) guidelines. A closed-loop cross-sequential design was used for quality control. The PROBAST (Prediction Model Risk of Bias Assessment Tool) and modified CLEAR (Checklist for Evaluation of Image-Based Artificial Intelligence Reports) benchmarks were used to systematically evaluate ROB and reporting quality.FindingsA total of 517 studies presenting 555 AI models were included and evaluated. Of these models, 461 (83.1%; 95% CI, 80.0%-86.2%) were rated as having a high overall ROB based on the PROBAST. The ROB was particular high in the analysis domain, including inadequate sample size (398 of 555 models [71.7%; 95% CI, 68.0%-75.6%]), poor model performance examination (with 100% of models lacking calibration examination), and lack of handling data complexity (550 of 555 models [99.1%; 95% CI, 98.3%-99.9%]). None of the AI models was perceived to be applicable to clinical practices. Overall reporting completeness (ie, number of reported items/number of total items) for the AI models was 61.2% (95% CI, 60.6%-61.8%), and the completeness was poorest for the technical assessment domain with 39.9% (95% CI, 38.8%-41.1%).Conclusions and RelevanceThis systematic review found that the clinical applicability and feasibility of neuroimaging-based AI models for psychiatric diagnosis were challenged by a high ROB and poor reporting quality. Particularly in the analysis domain, ROB in AI diagnostic models should be addressed before clinical application.

A new era in cognitive neuroscience: the tidal wave of artificial intelligence (AI)

“…Another notable challenge for prioritization is the poor generalizability of certain AI models. Despite the promising contributions of AI-assisted diagnostics in medicine, the current poor generalizability and applicability of the majority of existing medical AI models remains a significant challenge [ 14 ]. As such, building generalizable AI models to demonstrate the “intrinsic” neural mechanisms of human cognition is a milestone achievement in the interface of cognitive neuroscience and AI.…”

Updated Perspectives on the Neurobiology of Substance Use Disorders Using Neuroimaging

“…These data suggest that AI models need to be refined before use in clinical practice. 80 Overall, the success of AI use in clinical practice relies upon training machine learning with large datasets, which is where research efforts are currently focused. 81 There is reason to believe that with careful assessment, and risk of bias safeguards in place, AI will be a useful tool in clinical practice.…”

Transcriptomic and neuroimaging data integration enhances machine learning classification of schizophrenia

“…Using multimodal neuroimaging data, patients with SZ have been observed to display a wide range of abnormalities in brain morphology (Hulshoff Pol et al ., 2002 ; Liu et al ., 2020 ) as well as the structural and functional connectome (Cui et al ., 2019 ; Griffa et al ., 2019 ; Gao et al , 2023 ). Given such observed brain abnormalities, emerging research implements conventional machine learning (ML) or deep learning (DL) frameworks to distinguish SZ patients from healthy individuals, aiming to build up an objective, valid model that augments diagnosis or prognosis of the disorder in clinical practice (Gao et al , 2018 ; Sadeghi et al ., 2022 ; Chen et al ., 2023 ; Sui et al ., 2023 ). Despite these efforts, the extant methods have yet to achieve clinical applicability.…”