Robust is not necessarily reliable: From within-subjects fMRI contrasts to between-subjects comparisons

Abstract

Advances in cognitive and affective neuroscience come largely from within-subjects comparisons, in which the functional significance of neural activity is determined by contrasting two or more experimental conditions. Clinical and social neuroscience studies have attempted to leverage between-subject variability in such condition differences to better understand psychopathology and other individual differences. Shifting from within-to between-subjects comparisons requires that measures have adequate internal consistency to function as individual difference variables. This is particularly relevant for difference scores-which have lower reliability. The field has assumed reasonable internal consistency of neural measures based on consistent findings across studies (i.e., if a within-subject difference in neural activity is robust, then it must be reliable). Using one of the most common fMRI paradigms in the clinical neuroscience literature (i.e., a face- and shape-matching task), in a large sample of adolescents (N = 139) we replicate a robust finding: amygdala activation is greater for faces than shapes. Moreover, we demonstrate that the internal consistency of the amygdala in face and shape blocks was excellent (Spearman-Brown corrected reliability [SB] > .94). However, the internal consistency of the activation difference between faces and shapes was nearly zero (SB = -.06). This reflected the fact that the amygdala response to faces and shapes was highly correlated (r = .97) across individuals. Increased neural activation to faces versus shapes could not possibly function as an individual difference measure in these data-illustrating how neural activation can be robust within subjects, but unreliable as an individual difference measure. Strong and reproducible condition differences in neural activity are not necessarily well-suited for individual differences research-and neuroimaging studies should always report the internal consistency of, and correlations between, activations used in individual differences research.

Reliability of a functional magnetic resonance imaging task of emotional conflict in healthy participants

“…This is comparable to the ICCs of the behavioral data for the overall conditions. However, for reaction time, ICC for the incongruent versus congruent contrast was reduced relative to the overall conditions, further supporting findings of Infantolino et al () and Hedge et al (2018), that in addition to examining the reliability of neural measures, the behavioral measures should be assessed as well. Additionally, the fact that some conditions (e.g., all faces, congruent or incongruent) had a larger number of trials than other contrasts, such as conflict monitoring, needs to be considered; poor reliability in those trials may, at least in part, relate to low statistical power (Brown et al, ).…”

“…Thus, considering a trade-off between the complexity of the model, or contrast, and its interpretability is important when assessing test-retest reliability, especially when an intervention is introduced between scans. Direct comparison of active contrasts (e.g., incongruent minus congruent) showed poor reliability also in other neuroimaging studies (e.g., [Infantolino, Luking, Sauder, Curtin, & Hajcak, 2018]); however, main effects, or conditions, such as congruent or incongruent showed comparatively better reliability. Furthermore, observed differences in reliability may also be related to the nature of the task and more so to the similarity of its trials.…”

Event‐related potentials to threat of predictable and unpredictable shock

“…(see supporting information), in line with recent work has suggested that researchers report the reliability of both types of measures (Infantolino, Luking, Sauder, Curtin, & Hajcak, 2018). When selecting measurement indices of unpredictable threat processing, researchers may wish to take into account their reported reliabilities.…”

What is the test-retest reliability of common task-fMRI measures? New empirical evidence and a meta-analysis

“…Change scores, which are produced by subtracting two measures, will always have lower reliability than their constituent measures (29) . Currently, the majority of task-fMRI measures are based on contrasts between conditions (i.e., change scores), undermining their reliability (45) . However, the widespread use of contrasts in task-fMRI is largely a vestige of experimental cognitive neuroscience.…”