Powerful Moderator Variables in Behavioral Science? Don’t Bet on Them (Version 3)

Abstract

The current upsurge of interest in research replicability (and the exposure of many failures of reproducibility) has led to a much discussion about the possible role of statistical moderation (i.e., variable × variable interactions) in behavioral and social science. These interactions are so widespread and powerful, it is often argued, that we should hardly be surprised when attempts to reproduce important findings frequently lead to failure. Prior literature provides little empirical evidence about how common powerful moderation is. Using five large-scale behavioral research datasets we sought to shed light on the issue. The data reflected several thousand people engaging in a variety of behaviors over considerable periods of time, and included hundreds of demographic and psychological independent variables (IVs). For each outcome variable, we measured the interaction of every pair of IVs. Many IVs had sizable main effects on behavior, but interactions were usually very small in magnitude. While there is no doubt that interactions can occur in behavioral science contexts, the priors revealed here suggest that this should be postulated as a last, not a first resort.

Estimating group differences in network models using moderation analysis

“…However, I expect that the performance is similar for three or more groups, as long as the sample size in each group remains constant. The effect sizes for group differences (or moderation effects) are typically small in observational behavioral datasets (e.g., Sherman & Pashler, 2019;Chaplin, 1997;McClelland & Judd, 1993). The true effect sizes are therefore likely to be close to θ = 0.05 or θ = 0.15 for Gaussian and Ising models, respectively.…”

Seed and Soil: Psychological Affordances in Contexts Help to Explain Where Wise Interventions Succeed or Fail

“…These trials feature large samples of students within schools randomly assigned to condition (12,486 and 26,406, respectively), large numbers of school sites intentionally sampled to permit cross-site comparisons (65 and 21, with the latter further divided into 365 Race × First-Generation Status × College × Cohort groups), and preregistered hypotheses and analyses. Such precautions are necessary because heterogeneity findings can be unreliable (Bloom & Michalopoulos, 2013), especially with small samples (Sherman & Pashler, 2019). Moreover, each intervention was homogenously persuasive across sites, as assessed by manipulation checks.…”

Capturing interactions, correlations, fits, and transactions: A Person-Environment Relations Model

“…More recently, Sherman and Pashler (2019) estimated the magnitude of interactions between person and environment variables in five studies, with 42 to 542,700 effects per study. The averaged incremental Rs (computed across all possible models within each study) beyond the main effects of the person and environment variables ranged from .019 to .131 for those five studies.…”

“…For person-environment relations, previous empirical research has shown that distinguishing between 18 And unfortunately likely also even within this very chapter! 19 I thank Gabriela Blum for bringing up this idea in a personal email communication (May, 2019).…”