Fear is an evolutionary adaption to a hazardous environment, linked to numerous complex behavioral responses, e.g., the fight-or-flight response, suiting their respective environment. However, for the sake of experimental control, fear is mainly investigated under rather artificial laboratory conditions. The latter transform these evolutionary adaptions into artificial responses, like keystrokes. The immersive, multidimensional character of virtual reality (VR) enables realistic behavioral responses, overcoming aforementioned limitations. To investigate authentic fear responses from a holistic perspective, participants explored either a negative or a neutral VR cave. To promote real-life behavior, we built a physical replica of the cave, providing haptic sensations. Electrophysiological correlates of fear-related approach and avoidance tendencies, i.e., frontal alpha asymmetries (FAA) were evaluated. To our knowledge, this is the first study to simultaneously capture complex behavior and associated electrophysiological correlates under highly immersive conditions. Participants in the negative condition exhibited a broad spectrum of realistic fear behavior and reported intense negative affect as opposed to participants in the neutral condition. Despite these affective and behavioral differences, the groups could not be distinguished based on the FAAs for the greater part of the cave exploration. Taking the specific behavioral responses into account, the obtained FAAs could not be reconciled with well-known FAA models. Consequently, putting laboratory-based models to the test under realistic conditions shows that they may not unrestrictedly predict realistic behavior. As the VR environment facilitated non-mediated and realistic emotional and behavioral responses, our results demonstrate VR’s high potential to increase the ecological validity of scientific findings (video abstract: https://www.youtube.com/watch?v=qROsPOp87l4&feature=youtu.be).

Background: Biological psychiatry aims to understand mental disorders in terms of altered neurobiological pathways. However, for one of the most prevalent and disabling mental disorders, Major Depressive Disorder (MDD), patients only marginally differ from healthy individuals on the group-level. Whether Precision Psychiatry can solve this discrepancy and provide specific, reliable biomarkers remains unclear as current Machine Learning (ML) studies suffer from shortcomings pertaining to methods and data, which lead to substantial over- as well as underestimation of true model accuracy. Methods: Addressing these issues, we quantify classification accuracy on a single-subject level in N=1,801 patients with MDD and healthy controls employing an extensive multivariate approach across a comprehensive range of neuroimaging modalities in a well-curated cohort, including structural and functional Magnetic Resonance Imaging, Diffusion Tensor Imaging as well as a polygenic risk score for depression. Findings: Training and testing a total of 2.4 million ML models, we find accuracies for diagnostic classification between 48.1% and 62.0%. Multimodal data integration of all neuroimaging modalities does not improve model performance. Similarly, training ML models on individuals stratified based on age, sex, or remission status does not lead to better classification. Even under simulated conditions of perfect reliability, performance does not substantially improve. Importantly, model error analysis identifies symptom severity as one potential target for MDD subgroup identification. Interpretation: Although multivariate neuroimaging markers increase predictive power compared to univariate analyses, single-subject classification — even under conditions of extensive, best-practice Machine Learning optimization in a large, harmonized sample of patients diagnosed using state-of-the-art clinical assessments — does not reach clinically relevant performance. Based on this evidence, we sketch a course of action for Precision Psychiatry and future MDD biomarker research. Funding: The German Research Foundation, and the Interdisciplinary Centre for Clinical Research of the University of Münster.

Introduction:Statistical effect sizes are systematically overestimated in small samples, leading to poor generalizability and replicability of findings in all areas of research. Due to the large number of variables, this is particularly problematic in neuroimaging research. While cross-validation is frequently used in multivariate machine learning approaches to assess model generalizability and replicability, the benefits for mass-univariate brain analysis are yet unclear. We investigated the impact of cross-validation on effect size estimation in univariate voxel-based brain-wide associations, using body mass index (BMI) as an exemplary predictor.Methods:A total of n=3401 adults were pooled from three independent cohorts. Brain-wide associations between BMI and gray matter structure were tested using a standard linear mass-univariate voxel-based approach. First, a traditional non-cross-validated analysis was conducted to identify brain-wide effect sizes in the total sample (as an estimate of a realistic reference effect size). The impact of sample size (bootstrapped samples ranging from n=25 to n=3401) and cross-validation on effect size estimates was investigated across selected voxels with differing underlying effect sizes (including the brain-wide lowest effect size). Linear effects were estimated within training sets and then applied to unseen test set data, using 5-fold cross-validation. Resulting effect sizes (explained variance) were investigated.Results:Analysis in the total sample (n=3401) without cross-validation yielded mainly negative correlations between BMI and gray matter density with a maximum effect size of R2p=.036 (peak voxel in the cerebellum). Effects were overestimated exponentially with decreasing sample size, with effect sizes up to R2p=.535 in samples of n=25 for the voxel with the brain-wide largest effect and up to R2p=.429 for the voxel with the brain-wide smallest effect. When applying cross-validation, linear effects estimated in small samples did not generalize to an independent test set. For the largest brain-wide effect a minimum sample size of n=100 was required to start generalizing (explained variance >0 in unseen data), while n=400 were needed for smaller effects of R2p=.005 to generalize. For a voxel with an underlying null effect, linear effects found in non-cross-validated samples did not generalize to test sets even with the maximum sample size of n=3401. Effect size estimates obtained with and without cross-validation approached convergence in large samples.Discussion:Cross-validation is a useful method to counteract the overestimation of effect size particularly in small samples and to assess the generalizability of effects. Train and test set effect sizes converge in large samples which likely reflects a good generalizability for models in such samples. While linear effects start generalizing to unseen data in samples of n>100 for large effect sizes, the generalization of smaller effects requires larger samples (n>400). Cross-validation should be applied in voxel-based mass-univariate analysis to foster accurate effect size estimation and improve replicability of neuroimaging findings. We provide open-source python code for this purpose (https://osf.io/cy7fp/?view_only=a10fd0ee7b914f50820b5265f65f0cdb).

No abstract

No abstract