Small sample sizes reduce the replicability of task-based fMRI studies

Abstract

Despite a growing body of research suggesting that task-based functional magnetic resonance imaging (fMRI) studies often suffer from a lack of statistical power due to too-small samples, the proliferation of such underpowered studies continues unabated. Using large independent samples across eleven tasks, we demonstrate the impact of sample size on replicability, assessed at different levels of analysis relevant to fMRI researchers. We find that the degree of replicability for typical sample sizes is modest and that sample sizes much larger than typical (e.g., N = 100) produce results that fall well short of perfectly replicable. Thus, our results join the existing line of work advocating for larger sample sizes. Moreover, because we test sample sizes over a fairly large range and use intuitive metrics of replicability, our hope is that our results are more understandable and convincing to researchers who may have found previous results advocating for larger samples inaccessible.

Evaluation of the spatial variability in the major resting‐state networks across human brain functional atlases

“…Therefore, this study joins recent calls advocating the importance of large sample sizes for reproducible results in the neuroimaging field (Poldrack et al, 2017;Turner et al, 2018). Our results suggest that atlases derived from samples of at least 100 individuals are conducive to greater reproducibility of atlas-based RSN studies.…”

“…Despite progress, reproducibility remains one of the major concerns in neuroimaging (Poldrack et al, 2017;Turner, Paul, Miller, & Barbey, 2018). With regard to RSNs, their reproducibility can be influenced by parameters used to acquire the rs-fMRI data (Gordon et al, 2016), as well as interindividual variability (Braga & Buckner, 2017;Gordon et al, 2017).…”

Investigating GABA concentrations measured with macromolecule suppressed and unsuppressed MEGA‐PRESS MR spectroscopy and their relationship with BOLD responses in the occipital cortex

“…These results also highlight the problems inherent in studying associations between variables derived from low‐sensitivity methods, such as fMRI and MRS, in terms of the sample sizes required. For fMRI research, recent work has helped demonstrate the limits of making inferences from small samples, suggesting that larger groups (>100) than have normally been used are necessary for robust and replicable analyses . Equivalent studies for MRS are not yet available, but the meta‐analysis reported here helps illustrate the problem in this context.…”

“…For fMRI research, recent work has helped demonstrate the limits of making inferences from small samples, suggesting that larger groups (>100) than have normally been used are necessary for robust and replicable analyses. 30,31 Equivalent studies for MRS are not yet available, but the meta-analysis reported here helps illustrate the problem in this context. Adding the current results to five previous studies produced overall evidence for a negative association between GABA estimates and BOLD amplitudes (total sample size = 100).…”

The importance of diversity in cognitive neuroscience

“…5 The increasing push for larger sample sizes in cognitive neuroscience research compounds these recruitment issues. 6,7 Even larger samples are needed to have sufficient power to examine demographic factors using either between-group designs or as covariates in cognitive neuroscience studies. The historical homogeneity of research samples in science combined with current practical recruitment concerns leads to a vicious cycle: The lack of available data regarding the relationship between demographic variables and brain structure and function means there are few established models to test.…”