Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition

Abstract

Functional magnetic resonance imaging (fMRI) studiesofemotion, personality, and social cognition have drawn much attention in recent years, with high-profile studies frequently reporting extremely high (e.g., >.8) correlations between brain activation and personality measures. We show that these correlations are higher than should be expected given the (evidently limited) reliability of both fMRI and personality measures. The high correlations are all the more puzzling because method sections rarely contain much detail about how the correlations were obtained. We surveyed authors of 55 articles that reported findings of this kind to determine a few details on how these correlations were computed. More than half acknowledged using a strategy that computes separate correlations for individual voxels and reports means of only those voxels exceeding chosen thresholds. We show how this nonindependent analysis inflates correlations while yielding reassuring-looking scattergrams. This analysis technique was used to obtain the vast majority of the implausibly high correlations in our survey sample. In addition, we argue that, in some cases, other analysis problems likely created entirely spurious correlations. We outline how the data from these studies could be reanalyzed with unbiased methods to provide accurate estimates of the correlations in question and urge authors to perform such reanalyses. The underlying problems described here appear to be common in fMRI research of many kinds—not just in studies of emotion, personality, and social cognition.

Better P-curves: Making P-curve analysis more robust to errors, fraud, and ambitious P-hacking, a Reply to Ulrich and Miller (2015).

“…Vul, Harris, Winkielman, and Pashler (2009) found that about 40% of functional magnetic resonance imaging (fMRI) researchers reported results in their articles only for the voxel most correlated with a variable of interest (within a region of interest). This example satisfies all three conditions.…”

Multiple testing corrections, nonparametric methods, and random field theory

“…Over a decade later, and one "Voodoo correlations" (Vul et al, 2009) imbroglio and postmortem ichthyological fMRI study (Bennett et al, 2011) later, it seems everyone agrees that (a) correcting inferences for the search over the brain is essential and (b) such corrections are not consistently utilized in fMRI. Hopefully some historical perspective can strengthen the discipline's resolve to uphold good statistical practice.…”

The left hemisphere learns what is right: Hemispatial reward learning depends on reinforcement learning processes in the contralateral hemisphere

“…Conditions were compared using traditional t-tests implemented in SPM. Due to recent concerns (see Vul et al (2009)), displayed effect sizes were calculated by averaging extracted beta values over all voxels within a ROI.…”