A systematic review of Null Hypothesis Significance Testing, sample sizes, and statistical power in research using the Implicit Relational Assessment Procedure

Abstract

Following recent calls to examine the replicability of behavioral research, I examine sample sizes and statistical power, key determinants of replicability, in research using a task that has seen broad use in behavioral research: the Implicit Relational Assessment Procedure. A systematic review was used to gather all published studies employing the IRAP and extract their designs and sample sizes. The use of Null Hypothesis Significance Testing was found to be nearly ubiquitous, justifying the examination of statistical power. Using an established method, median sample sizes were used to estimate the statistical power to detect the average published effect size in psychological research (r = .20) in each year. Sample sizes and the statistical power they imply were found to be very low in IRAP studies (in 2022, median N = 64, power = .34). A the current rate of growth, power will only reach the recommended minimum of .80 by 2080. The IRAP literature was directly compared with the Social and Personality psychology literature using an existing dataset. Median sample sizes and their implied statistical power were lower in the IRAP literature in all years than they were in Social and Personality psychology at the beginning of the Replication Crisis in 2011 and in all subsequent years. Improvements in sample sizes and statistical power in the Social and Personality psychology literature were significantly and substantially larger than in the IRAP literature. Direct tests of the reproducibility and replicability of claims in the IRAP literature are needed.