Predicting the future of neuroimaging predictive models in mental health

Abstract

Predictive modeling using neuroimaging data has the potential to improve our understanding of the neurobiology underlying psychiatric disorders and putatively information interventions. Accordingly, there is a plethora of literature reviewing published studies, the mathematics underlying machine learning, and the best practices for using these approaches. As our knowledge of mental health and machine learning continue to evolve, we instead aim to look forward and “predict” topics that we believe will be important in current and future studies. Some of the most discussed topics in machine learning, such as bias and fairness, the handling of dirty data, and interpretable models, may be less familiar to the broader community using neuroimaging-based predictive modeling in psychiatry. In a similar vein, transdiagnostic research and targeting brain-based features for psychiatric intervention are modern topics in psychiatry that predictive models are well-suited to tackle. In this work, we target an audience who is a researcher familiar with the fundamental procedures of machine learning and who wishes to increase their knowledge of ongoing topics in the field. We aim to accelerate the utility and applications of neuroimaging-based predictive models for psychiatric research by highlighting and considering these topics. Furthermore, though not a focus, these ideas generalize to neuroimaging-based predictive modeling in other clinical neurosciences and predictive modeling with different data types (e.g., digital health data).

Power and reproducibility in the external validation of brain-phenotype predictions

“…In addition, external validation may help to ameliorate machine learning ethical issues (Mitchell et al ., 2019; Chandler, Foltz and Elvevåg, 2020), including bias (Benkarim et al ., 2021; Greene et al ., 2022; Li et al ., 2022) and trustworthiness (Rosenblatt et al ., 2023). For bias, evaluating models in external datasets will better depict the robustness and generalizability of brain-phenotype associations in populations with different characteristics (Mehrabi et al ., 2021; Tejavibulya et al ., 2022). For trustworthiness, external validation ensures that data manipulations are not driving the results (Finlayson et al ., 2019; Rosenblatt et al ., 2023).…”

Generalizability of Clinical Prediction Models in Mental Health - Real-World Validation of Machine Learning Models for Depressive Symptom Prediction

“…Systematic differences in available data sources and sampling effects between real-world clinical populations and those derived from research cohorts are thought to hinder generalizability of machine learning models in mental health (2)(3)(4). Clinical and demographic differences between and within research and real-world samples may lead to heterogeneity, which substantially impairs prediction accuracy and model generalizability (5).…”

“…Clinical and demographic differences between and within research and real-world samples may lead to heterogeneity, which substantially impairs prediction accuracy and model generalizability (5). Assessing model generalization on real-world data is critical as they represent the populations for which predictions are intended, thus minimizing bias (3). While successful attempts have been made to train models for clinically relevant predictions within a single research dataset (6)(7)(8), previous investigations have often overlooked external validation, specifically validation in real-world clinical samples (9).…”

The effects of data leakage on connectome-based machine learning models

“…Understanding individual differences in brain-behavior relationships is a central goal of neuroscience. As part of this goal, machine learning approaches using neuroimaging data, such as functional connectivity, have grown increasingly popular in predicting numerous phenotypes 1 , including cognitive performance 2–6 , age 7–10 , and several clinically-relevant outcomes 11–13 . Compared to classic statistical inference, prediction offers advantages in replicability and generalizability, as it evaluates models on participants unseen during model training 14,15 .…”