Clinical prediction models in psychiatry: a systematic review of two decades of progress and challenges

Abstract

Recent years have seen the rapid proliferation of clinical prediction models aiming to support risk stratification and individualized care within psychiatry. Despite growing interest, attempts to synthesize current evidence in the nascent field of precision psychiatry have remained scarce. This systematic review therefore sought to summarize progress towards clinical implementation of prediction modeling for psychiatric outcomes. We searched MEDLINE, PubMed, Embase, and PsychINFO databases from inception to September 30, 2020, for English-language articles that developed and/or validated multivariable models to predict (at an individual level) onset, course, or treatment response for non-organic psychiatric disorders (PROSPERO: CRD42020216530). Individual prediction models were evaluated based on three key criteria: (i) mitigation of bias and overfitting; (ii) generalizability, and (iii) clinical utility. The Prediction model Risk Of Bias ASsessment Tool (PROBAST) was used to formally appraise each study’s risk of bias. 228 studies detailing 308 prediction models were ultimately eligible for inclusion. 94.5% of developed prediction models were deemed to be at high risk of bias, largely due to inadequate or inappropriate analytic decisions. Insufficient internal validation efforts (within the development sample) were also observed, while only one-fifth of models underwent external validation in an independent sample. Finally, our search identified just one published model whose potential utility in clinical practice was formally assessed. Our findings illustrated significant growth in precision psychiatry with promising progress towards real-world application. Nevertheless, these efforts have been inhibited by a preponderance of bias and overfitting, while the generalizability and clinical utility of many published models has yet to be formally established. Through improved methodological rigor during initial development, robust evaluations of reproducibility via independent validation, and evidence-based implementation frameworks, future research has the potential to generate risk prediction tools capable of enhancing clinical decision-making in psychiatric care.

Predicting suicide risk in 137 112 people with severe mental illness in Finland: external validation of the Oxford Mental Illness and Suicide tool (OxMIS)

“…Furthermore, a recent systematic review of clinical prediction models in psychiatry found that only 16% of the studies had been validated in wholly independent samples and reported measures of discrimination in development and validation samples. 5 Importantly, the same review found that nearly four out of ve (78%) of these studies reported poorer out-of-sample discrimination. In contrast, we found that OxMIS maintained a similar performance from its Swedish validation, with no clear differences between the reported AUCs (range of 0.70-0.71).…”

“…Part of the reason for this is that prediction models in mental health are very rarely externally validated. 5 Such validation is a necessary step on the path to implementation alongside work on feasibility, acceptability and clinical impact. In the suicide eld, this is not different -few models have been externally validated despite their clinical use in some settings.…”

Postpartum depression: a developed and validated model predicting individual risk in new mothers

“…However, we strongly emphasize multiple steps are needed before our model or similar work can be implemented in real-world clinical routines. We note that this is a challenge not only related to PPD prediction, as Meehan et al recently found only one out of 308 published prediction models within psychiatry was formally evaluated and assessed for usefulness in clinical care [ 53 ]. We propose that moving forward any efforts toward clinical implementation could include engagement with stakeholders, including patients, clinicians, and politicians, to evaluate how to maximize the translational potential of our model as well as models from other groups.…”

Computational Psychiatry and Computational Neurology: Seeking for Mechanistic Modeling in Cognitive Impairment and Dementia

“…When interpreting models, researchers need to be conscious of potential challenges. For instance, predictive models in psychiatry still suffer from overfitting and lack of generalisability and validation (Meehan et al, 2022 ). Finding a model with the appropriate complexity, therefore, requires finding a suitable balance between bias and variance (Lever et al, 2016 ).…”