Multiple Holdouts With Stability: Improving the Generalizability of Machine Learning Analyses of Brain–Behavior Relationships

Abstract

BACKGROUND: In 2009, the National Institute of Mental Health launched the Research Domain Criteria, an attempt to move beyond diagnostic categories and ground psychiatry within neurobiological constructs that combine different levels of measures (e.g., brain imaging and behavior). Statistical methods that can integrate such multimodal data, however, are often vulnerable to overfitting, poor generalization, and difficulties in interpreting the results. METHODS: We propose an innovative machine learning framework combining multiple holdouts and a stability criterion with regularized multivariate techniques, such as sparse partial least squares and kernel canonical correlation analysis, for identifying hidden dimensions of cross-modality relationships. To illustrate the approach, we investigated structural brain-behavior associations in an extensively phenotyped developmental sample of 345 participants (312 healthy and 33 with clinical depression). The brain data consisted of whole-brain voxel-based gray matter volumes, and the behavioral data included item-level self-report questionnaires and IQ and demographic measures. RESULTS: Both sparse partial least squares and kernel canonical correlation analysis captured two hidden dimensions of brain-behavior relationships: one related to age and drinking and the other one related to depression. The applied machine learning framework indicates that these results are stable and generalize well to new data. Indeed, the identified brain-behavior associations are in agreement with previous findings in the literature concerning age, alcohol use, and depression-related changes in brain volume. CONCLUSIONS: Multivariate techniques (such as sparse partial least squares and kernel canonical correlation analysis) embedded in our novel framework are promising tools to link behavior and/or symptoms to neurobiology and thus have great potential to contribute to a biologically grounded definition of psychiatric disorders.

Associations between aversive learning processes and transdiagnostic psychiatric symptoms in a general population sample

“…method has been employed successfully in prior studies to provide insight into how panels of cognitive and behavioural measures relate to multivariate neuroimaging-derived phenotypes [34][35][36] . PLS-like analyses can be problematic if not properly validated (for example producing spurious results due to overfitting), and so we adopted best-practice methods for validating these results [37][38][39] , selecting the optimal number of components using cross-validation and training the model on 75% of the data, before testing its performance on the remaining 25% of the data.…”

How to Determine the Early Warning Threshold Value of Meteorological Factors on Influenza through Big Data Analysis and Machine Learning

“…Generally speaking, the sample size included in D 1 should account for at least 2/3 of the D of the entire dataset. In practice, there is a widely used Hold-Out method [ 22 ]: when the data has obvious time series factors, the time of online data is after the offline dataset. In this case, the training set and test set should be divided according to time.…”

Probing the clinical and brain structural boundaries of bipolar and major depressive disorder

“…For each permuted sample, we performed s-PLS-DA using the optimal parameters identified in the original model and retested the weights identified in the permuted data on the held-out set. To test generalizability, we followed an established multi-holdout procedure 33 , whereby we created ten randomly selected optimization/held-out samples using the 80/20 split and repeated the above analyses. Of the ten samples, the one yielding the lowest P -value in the held-out set was chosen as the best model based on a Bonferroni corrected P < 0.05.…”