Reproducibility of R-fMRI Metrics on the Impact of Different Strategies for Multiple Comparison Correction and Sample Sizes

Abstract

Concerns regarding reproducibility of resting-state functional magnetic resonance imaging (R-fMRI) findings have been raised. Little is known about how to operationally define R-fMRI reproducibility and to what extent it is affected by multiple comparison correction strategies and sample size. We comprehensively assessed two aspects of reproducibility, test-retest reliability and replicability, on widely used R-fMRI metrics in both between-subject contrasts of sex differences and within-subject comparisons of eyes-open and eyes-closed (EOEC) conditions. We noted permutation test with Threshold-Free Cluster Enhancement (TFCE), a strict multiple comparison correction strategy, reached the best balance between family-wise error rate (under 5%) and test-retest reliability / replicability (e.g., 0.68 for test-retest reliability and 0.25 for replicability of amplitude of low-frequency fluctuations (ALFF) for between-subject sex differences, 0.49 for replicability of ALFF for within-subject EOEC differences). Although R-fMRI indices attained moderate reliabilities, they replicated poorly in distinct datasets (replicability < 0.3 for between-subject sex differences, < 0.5 for within-subject EOEC differences). By randomly drawing different sample sizes from a single site, we found reliability, sensitivity and positive predictive value (PPV) rose as sample size increased. Small sample sizes (e.g., < 80 (40 per group)) not only minimized power (sensitivity < 2%), but also decreased the likelihood that significant results reflect "true" effects (PPV < 0.26) in sex differences. Our findings have implications for how to select multiple comparison correction strategies and highlight the importance of sufficiently large sample sizes in R-fMRI studies to enhance reproducibility.. CC-BY 4.0 International license peer-reviewed) is the author/funder. It is made available under a

Shifted balance of dorsal versus ventral striatal communication with frontal reward and regulatory regions in cannabis‐dependent males

“…To determine the shift in ventral vs dorsal striatal networks associated with dependence, the subjectlevel seed-specific Z-maps were submitted to a two-way mixed ANOVA with seed (VS and DS) entered as a within-subject factor and group (controls and cannabis users) entered as a betweensubject factor employed in the FSL's Randomise Tool. Corrections for multiple comparisons were conducted using permutation-based inferences (10,000 permutations) with Threshold-Free Clustering Enhancement (TFCE) which provides a strict control while improving replicability (Chen, Lu, & Yan, 2018;Smith & Nichols, 2009…”

Reduced Default Mode Network Functional Connectivity in Patients with Recurrent Major Depressive Disorder

“…The pathophysiology of MDD remains unknown despite intensive efforts, including neuroimaging studies. However, the small sample size of most MDD neuroimaging studies entails low sensitivity and reliability (2,3). An exception is the Enhancing NeuroImaging Genetics through Meta-Analysis (ENIGMA) consortium which meta-and mega-analyzed thousands of structural MRI scans from MDD patients and healthy controls (4,5).…”

Aberrant intrinsic functional connectivity in thalamo‐cortical networks in major depressive disorder

“…With this setup, we can achieve a two-tailed voxel P < 0.001 and cluster P < 0.05 and well control the family-wise error rate under 5%. 25,26…”