Power and reproducibility in the external validation of brain-phenotype predictions

Abstract

Identifying reproducible and generalizable brain-phenotype associations is a central goal of neuroimaging. Consistent with this goal, prediction frameworks evaluate brain-phenotype models in unseen data. Most prediction studies train and evaluate a model in the same dataset. However, external validation, or the evaluation of a model in an external dataset, provides a better assessment of robustness and generalizability. Despite the promise of external validation and calls for its usage, the statistical power of such studies has yet to be investigated. In this work, we ran over 60 million simulations across several datasets, phenotypes, and sample sizes to better understand how the sizes of the training and external datasets affect statistical power. We found that prior external validation studies used sample sizes prone to low power, which may lead to false negatives and effect size inflation. Furthermore, increases in the external sample size led to increased simulated power directly following theoretical power curves, whereas changes in the training dataset size offset the simulated power curves. Finally, we compared the performance of a model within a dataset to the external performance. The within-dataset performance was typically withinr=0.2of the cross-dataset performance, which could help decide how to power future external validation studies. Overall, our results illustrate the importance of considering the sample sizes of both the training and external datasets when performing external validation.

Brain-phenotype predictions can survive across diverse real-world data

“…This approach includes using large sample sizes to create and externally validate models. In contrast to most studies using external validation, the sample sizes for external validation were of the same order as the training data (Rosenblatt et al, 2023; Yeung et al, 2022). In fact, given that two external datasets were used to validate each model, more data was used to test a model than train it.…”

“…First, we used three large developmental datasets to maximize statistical power. Few large-scale neuroimaging studies incorporate any form of external validation (Rosenblatt et al, 2023; Yeung et al, 2022). In addition to internal cross-validation, each model was validated in two independent large-scale datasets.…”

“…This approach includes using large sample sizes to create and externally validate models. In contrast to most studies using external validation, the sample sizes for external validation were of the same order as the training data (Rosenblatt et al, 2023; . CC-BY-ND 4.0 International license available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.…”

“…Only a minority of neuroimaging studies undertake any form of external validation. Among those that do, the median external sample size is only n=108 and is underpowered in most cases (Rosenblatt et al, 2023; Yeung et al, 2022). Further, real-world and eventual clinical applications demand not only external validation but also generalization across different imaging and phenotypic features (Dockès et al, 2021; Woo et al, 2017).…”