A Systematic Evaluation of Machine Learning-based Biomarkers for Major Depressive Disorder across Modalities

Abstract

Background: Biological psychiatry aims to understand mental disorders in terms of altered neurobiological pathways. However, for one of the most prevalent and disabling mental disorders, Major Depressive Disorder (MDD), patients only marginally differ from healthy individuals on the group-level. Whether Precision Psychiatry can solve this discrepancy and provide specific, reliable biomarkers remains unclear as current Machine Learning (ML) studies suffer from shortcomings pertaining to methods and data, which lead to substantial over- as well as underestimation of true model accuracy. Methods: Addressing these issues, we quantify classification accuracy on a single-subject level in N=1,801 patients with MDD and healthy controls employing an extensive multivariate approach across a comprehensive range of neuroimaging modalities in a well-curated cohort, including structural and functional Magnetic Resonance Imaging, Diffusion Tensor Imaging as well as a polygenic risk score for depression. Findings: Training and testing a total of 2.4 million ML models, we find accuracies for diagnostic classification between 48.1% and 62.0%. Multimodal data integration of all neuroimaging modalities does not improve model performance. Similarly, training ML models on individuals stratified based on age, sex, or remission status does not lead to better classification. Even under simulated conditions of perfect reliability, performance does not substantially improve. Importantly, model error analysis identifies symptom severity as one potential target for MDD subgroup identification. Interpretation: Although multivariate neuroimaging markers increase predictive power compared to univariate analyses, single-subject classification — even under conditions of extensive, best-practice Machine Learning optimization in a large, harmonized sample of patients diagnosed using state-of-the-art clinical assessments — does not reach clinically relevant performance. Based on this evidence, we sketch a course of action for Precision Psychiatry and future MDD biomarker research. Funding: The German Research Foundation, and the Interdisciplinary Centre for Clinical Research of the University of Münster.

Harmonization techniques for machine learning studies using multi-site functional MRI data

“…The results show that for all representation-dataset configurations, no harmonization method outperformed the baseline more than one accuracy point. Note that our results are generally on-par with the current reported state-of-the art performances for both datasets [37,31,38,25,39,40,41,42].…”

Lower motor performance is linked with poor sleep quality, depressive symptoms, and grey matter volume alterations

“…This potentially indicates differential modes of association with MP. A possible explanation for these findings is the lack of a common link between brain structure and sleep quality/depressive symptoms, as highlighted previously (Olfati et al, 2024;Weihs et al, 2023;Winter et al, 2024).…”

“…Large-scale studies identified a link between longer sleep duration and higher GMV in basal ganglia but have failed to establish significant associations between other sleep health dimensions and GMV, such as insomnia (Schiel et al, 2023;Weihs et al, 2023). Similarly, investigations on depressive symptoms revealed either minimal effects or no significant associations between depressive symptoms/clinical depression and brain structure (for the ENIGMA-Major Depressive Disorder Working Group et al, 2016Olfati et al, 2024;Winter et al, 2022Winter et al, , 2024. Taken together, the neurobiological correlates of the interplay between sleep quality and depressive symptoms with MP remain unclear.…”

Accurate Machine Learning Prediction in Psychiatry Needs the Right Kind of Information

“…To address this, prediction models should incorporate (1) personalized data from each individual, (2) sources of data clinicians use, and (3) valid measures at the individual level. In future research, it will also be important to focus on the other side of the equation as well because models in psychiatry are often specified to predict binary outcomes for disorders (eg, depressed vs control) that are known to have questionable validity and reliability . Although continued empirical and conceptual work in this area will be necessary, we believe that the field must prioritize providing ML models with substantially more data for each individual while also remaining mindful of the challenges that come with increased dimensionality.…”

“…Given the inadequate performance of ML models in psychiatric contexts to date, it is important to consider whether ML models have sufficient information to generate accurate predictions. Herein, we discuss the importance of ample information for model performance and contrast prediction in psychiatry with successes in other fields.…”