p-Curve Analysis of the Taylor Aggression Paradigm: Estimating Evidentiary Value and Statistical Power Across 50 Years of Research

Abstract

The overall reliability or evidentiary value of any body of literature is established in part by ruling out publication bias for any observed effects. Questionable research practices have potentially undermined the evidentiary value of commonly-used research paradigms in psychological science. Subsequently, the evidentiary value of these common methodologies remains uncertain. To quantify the severity of these issues in the literature, we selected the Taylor Aggression Paradigm (TAP) as a case study and submitted 170 hypothesis-tests spanning over 50 years of research to a preregistered p-curve analysis. The TAP literature (N = 24,685) demonstrated significant evidentiary value but yielded a small average effect size (d = .29) and inadequate power (38%). Main effects demonstrated greater evidentiary value, power, and effect sizes than interactions. Studies that tested the effects of measured traits did not differ in evidentiary value or power to those that tested the effects of experimentally-manipulated states. Exploratory analyses revealed that evidentiary value, statistical power, and effect sizes have improved over time. We provide recommendations for researchers who seek to maximize the evidentiary value of their psychological measures.

An empirically based power primer for laboratory aggression research

“…Similarly, the same pattern of power across predictor variable correlations was observed such that across effect size estimates and most sample sizes, statistical power decreased somewhat as the correlations between the predictor variables increased. Researchers who expect to locate an interaction effect of a magnitude typically found in this literature (i.e., r = ~.01; West et al, 2020) using a two‐tailed test should plan to collect at least N = 100,000 to achieve ~80% power across all tested predictor variable correlations. Even if researchers assume a three‐way interaction effect size of a magnitude five times larger than the size typically observed, they should plan to collect at least N = 5000 to achieve ~80% power.…”

“…Researchers who expect to locate an interaction effect of a magnitude typically found in this literature (i.e., r = ~0.05; West et al, 2020) with a two‐tailed test should plan to collect at least N = 5000 to achieve ~80% power across all tested predictor variable correlations. Even if researchers assume a two‐way interaction effect size of a magnitude of double the size typically found, they should plan to collect at least N = 1000 to achieve ~80% power across all tested predictor variable correlations.…”

“…None of the 55 reviewed studies reported sample sizes that met this benchmark, which is not surprising given the time and resource intensive nature of laboratory aggression work. Similarly, if one expects a three‐way interaction of the magnitude ( r = .01) derived from the data and coding scripts provided by West and colleagues (2020), a sample of N = 100,000 is needed to achieve >80% power with a two‐tailed test, a benchmark that none of the reviewed studies approach. The practical upshot of this finding is that we urge lab aggression researchers to consider an indefinite moratorium on testing three‐way interactions and only propose them in cases where a massive interactive effect can be expected.…”

“…For the main effect size estimates in the interaction analyses, r = .20 was used for the main effect of a continuous predictor based on the meta‐analysis by Hyatt et al (2019). Similarly, based on the results of the work by West and colleagues (2020), r = .10 was used for the main effect of condition as the best estimate of the effect of condition after correcting for publication bias. Finally, in a large, internal dataset ( N = 1,790; Lasko & Chester, 2021), the gender difference in TAP scores was d = 0.23 ( r = .11) such that males exhibited higher aggression scores than females.…”

“…Several recent reviews suggest that power is a substantial concern in the laboratory aggression literature, where the largest meta‐analytic effect sizes (i.e., psychopathy and lab aggression) are in the realm of r = ~.20 (Hyatt, Zeichner, et al, 2019). In a p ‐curve analysis (Simonsohn et al, 2014a) spanning the entirety of studies using the seminal Taylor Aggression Paradigm (TAP; Taylor, 1967) 1 , the mean effect size for hypothesized main effects (e.g., effect of an individual difference variable or experimental condition) across all studies was Cohen's d = 0.37 (or r = .18) and d = 0.10 (or r = .05) for interaction effects (West et al, 2020). Of note, this average main effect size is consistent with the average effect size ( r = .20) found in the recent replication effort by the Open Science Collaboration.…”