Meta-analyses are no substitute for registered replications: a skeptical perspective on religious priming

Abstract

According to a recent meta-analysis, religious priming has a positive effect on prosocial behavior (Shariff et al., 2015). We first argue that this meta-analysis suffers from a number of methodological shortcomings that limit the conclusions that can be drawn about the potential benefits of religious priming. Next we present a re-analysis of the religious priming data using two different meta-analytic techniques. A Precision-Effect Testing–Precision-Effect-Estimate with Standard Error (PET-PEESE) meta-analysis suggests that the effect of religious priming is driven solely by publication bias. In contrast, an analysis using Bayesian bias correction suggests the presence of a religious priming effect, even after controlling for publication bias. These contradictory statistical results demonstrate that meta-analytic techniques alone may not be sufficiently robust to firmly establish the presence or absence of an effect. We argue that a conclusive resolution of the debate about the effect of religious priming on prosocial behavior – and about theoretically disputed effects more generally – requires a large-scale, preregistered replication project, which we consider to be the sole remedy for the adverse effects of experimenter bias and publication bias.

Why Are Protestants More Prosocial Than Catholics? A Comparative Study Among Orthodox Dutch Believers

“…This finding echoes more recent concerns, indicating that the observed relation between religiousness and prosociality may be restricted to self-report measures rather than actual behavioral measures (Galen, 2012;Shariff, 2015). Although it has been argued that experimental techniques such as religious priming might provide a solution to this problem (Willard, Shariff, & Norenzayan, 2016), methodological concerns (i.e., researcher and publication bias) severely limit the conclusions that can be drawn based on these studies (van Elk et al, 2015). Still, we note that the directionality of the effects that we reported is comparable to previous studies in which actual behavior (e.g., actual donations to religious charities) was measured (Bekkers & Schuyt, 2008;Bekkers & Wiepking, 2011b)-thereby increasing the confidence in the ecological validity of our measurements.…”

Parafoveal preview effects from word N + 1 and word N + 2 during reading: A critical review and Bayesian meta-analysis

“…Although the robustness analysis showed that this had little effect on the present results, it does illustrate very well the need for high-power N + 2 studies in the future. Although meta-analysis is a good tool for summarizing evidence, it is no substitute for replication studies (Van Elk et al, 2015). In this sense, we recommend that high-power, registered replications of N + 2 studies be undertaken in the future.…”

A meta-analysis of the survival-processing advantage in memory

“…van Elk et al (2015) point out that conclusions from meta-analyses are often limited, due to methodological shortcomings. Bias-correction techniques often can be inconsistent across a range of conditions, requiring ranges of effect size interpretations instead of the use of a single technique (Inzlicht et al, 2015).…”