Are Small Effects the Indispensable Foundation for a Cumulative Psychological Science? A Reply to Götz et al. (2022)

Abstract

Götz et al. (2021) argue that small effects are the indispensable foundation for a cumulative psychological science. Whilst we applaud their efforts to bring this important discussion to the forefront, we argue that their core arguments do not hold up under scrutiny, and if left uncorrected have the potential to undermine best practices in reporting and interpreting effect size estimates. Their article can be used as a convenient blanket defense to justify ‘small’ effects as meaningful. In our reply, we first argue that comparisons between psychological science and genetics are fundamentally flawed because these disciplines have vastly different goals and methodology. Second, we argue that p-values, not effect sizes, are the main currency for publication in psychology, meaning that any biases in the literature are caused by this pressure to publish statistically significant results, not a pressure to publish large effects. Third, we contend that claims regarding small effects as important and consequential must be supported by empirical evidence, or at least require a falsifiable line of reasoning. Finally, we propose that researchers should evaluate effect sizes in relative, not absolute terms, and provide several approaches of how this can be achieved.

Revisiting the rigidity-of-the-right hypothesis: A meta-analytic review.

“…Given that many scholars have deemed effects of these magnitudes to possess theoretical and practical importance for other notable predictors of social conservatism (e.g., low educational attainment; Lipset, 1959;Schoon et al, 2010), it seems reasonable, at first blush, to conclude that rigidity is meaningfully related to social conservatism. Moreover, even certain "small" effects can have profound real-world implications (Abelson, 1985;Funder & Ozer, 2019) and some authors posit that, like genetics, psychology comprises networks of complex phenomena that are an outcome of the additive influence of small effects, which may be meaningful in aggregate and/or longitudinally (Gotz et al, 2022;Primbs et al, 2022; see Supplemental File 3 for an in-depth discussion of our effect size magnitudes).…”

The inutility of studying fictional-coin giving for understanding the relationship between attachment and prosocial development

“…In the present case, attachment was interpreted by van Lange as a temporally distant childhood factor (just like the number of siblings one grew up with), so its effects are unlikely to cumulate further. Furthermore, in large enough datasets in psychology, all variables will correlate with each other, which is also known as the "crud" factor (Meehl, 1990;Primbs et al, 2022;Vul et al, 2009). The crud factor in psychology can be as large as r = 0.10 (Ferguson & Heene, 2021), thus already larger than all of the effects observed here.…”

What are small, medium and large effect sizes for exercise treatments of tendinopathy? A systematic review and meta-analysis

“…49 In addition to challenges associated with estimates, the present analysis is limited in the ability to address clinical significance. While benchmarking effect sizes using empirical values within a specific context is important, 63 the labelling of thresholds as small, medium and large remains somewhat arbitrary. In contrast, anchor-based approaches that use an external criterion such as global rating of change are viewed as a superior method to establish important thresholds such as the MIC.…”