False-positive neuroimaging: Undisclosed flexibility in testing spatial hypotheses allows presenting anything as a replicated finding

Abstract

Hypothesis testing in neuroimaging studies relies heavily on treating named anatomical regions (e.g., "the amygdala") as unitary entities. Though data collection and analyses are conducted at the voxel level, inferences are often based on anatomical regions. The discrepancy between the unit of analysis and the unit of inference leads to ambiguity and flexibility in analyses that can create a false sense of reproducibility. For example, hypothesizing effects on "amygdala activity" does not provide a falsifiable and reproducible definition of precisely which voxels or which patterns of activation should be observed. Rather, it comprises a large number of unspecified sub-hypotheses, leaving room for flexible interpretation of findings, which we refer to as "model degrees of freedom." From a survey of 135 functional Magnetic Resonance Imaging studies in which researchers claimed replications of previous findings, we found that 42.2% of the studies did not report any quantitative evidence for replication such as activation peaks.Only 14.1% of the papers used exact coordinate-based or a priori pattern-based models.Of the studies that reported peak information, 42.9% of the 'replicated' findings had peak coordinates more than 15 mm away from the 'original' findings, suggesting that different brain locations were activated, even when studies claimed to replicate prior results. To reduce the flexible and qualitative region-level tests in neuroimaging studies, we recommend adopting quantitative spatial models and tests to assess the spatial reproducibility of findings. Techniques reviewed here include permutation tests on peak distance, Bayesian MANOVA, and a priori multivariate pattern-based models. These practices will help researchers to establish precise and falsifiable spatial hypotheses, promoting a cumulative science of neuroimaging.

Interactions Between Methodological and Interindividual Variability: How Monetary Incentive Delay (MID) Task Contrast Maps Vary and Impact Associations with Behavior.

“…One approach may be to use parametric modulators, which has been used in prior analyses, but is largely underutilized (Aloi et al, 2019;Joseph et al, 2016). In addition to improving estimates of functional parcels (Nikolaidis et al, 2020), multivariate pattern analyses may help with the reproducibility of theorized cognitive processes (Hong et al, 2019). Multivariate, cross-validated, pattern analyses can provide a priori activation patterns and locations that can be confirmed out of sample, reducing the possibility of exploring multiple hypotheses.…”

“…Recent evidence has suggested that analytic methods and decisions may not only alter outcomes, but also result in different interpretations of fMRI analyses. For instance, Hong et al (2019) found that peak level coordinates from various studies have a high degree of variability, that may often lead to inaccurate conclusions about replication. Although activations may be close in distance between two groups (or studies), such that they appear to be in similar brain regions, these may not be related to a 'replication' of a neural process that is hypothesized, due to a lack of neural specificity.…”

Replicable brain signatures of emotional bias and memory based on diffusion kurtosis imaging of white matter tracts

“…Further, the significant replication of results, despite the age mismatch between the Discovery and Replication cohorts, may suggest that the observed effects are robust to differences in age. Our approach using a specific, pre‐defined composite cluster was not subject to the “model degrees of freedom” which purportedly have driven false‐positive replications in neuroimaging (Hong, Yoo, Han, Wager, & Woo, ). Last, our study did not make use of multivariate data mining or machine learning approaches which are gaining popularity with application in prediction studies, and show high levels of accuracy (Benedict et al, ; Dyrba et al, ; Moradi, Pepe, Gaser, Huttunen, & Tohka, ).…”

The case for preregistering all region of interest (ROI) analyses in neuroimaging research

“…Frequently, the rationale supporting ROI selection is based on vague statements or on citing previous findings without motivating how these suggest an association between a well-defined brain area and a specific mental process. Across the neuroimaging literature, it is unclear how many published papers rely on an adequate definition of ROIs (Hong, Yoo, Han, Wager, & Woo, 2019). Hypothetically, the choice of any ROI may be justified based on the existing literature, leading to a "garden of forking paths" of potential data-dependent analyses (Gelman & Loken, 2014).…”

“…Useful tools in this regard are represented by GingerALE (Müller et al, 2018) and Seed-based d Mapping (SDM; Albajes-Eizagirre, Solanes, Vieta, & Radua, 2019), which quantitatively aggregate neuroimaging findings, as well as by Neurosynth (Yarkoni, Poldrack, Nichols, Van Essen, & Wager, 2011) and NeuroQuery (Dockès et al, 2020), which estimate the association between voxels and terms semantically related to the study hypothesis or to functional areas of interest (e.g., fusiform face area). The definition of ROIs could also be based on inherently vague anatomical terms (e.g., medial prefrontal cortex) or it may refer to a large patch of the cortex (e.g., precise location within the Superior Frontal Gyrus; Hong et al, 2019). Interestingly, Neurosynth and Neuroquery can be used to precisely define anatomical ROIs, which have no clear boundaries or are typically not included in anatomical atlases, as the temporo-parietal junction (Lettieri et al, 2019).…”