Many Labs 5: Registered Multisite Replication of the Tempting-Fate Effects in Risen and Gilovich (2008)

Abstract

Risen and Gilovich (2008) found that subjects believed that “tempting fate” would be punished with ironic bad outcomes (a main effect), and that this effect was magnified when subjects were under cognitive load (an interaction). A previous replication study (Frank & Mathur, 2016) that used an online implementation of the protocol on Amazon Mechanical Turk failed to replicate both the main effect and the interaction. Before this replication was run, the authors of the original study expressed concern that the cognitive-load manipulation may be less effective when implemented online than when implemented in the lab and that subjects recruited online may also respond differently to the specific experimental scenario chosen for the replication. A later, large replication project, Many Labs 2 (Klein et al. 2018), replicated the main effect (though the effect size was smaller than in the original study), but the interaction was not assessed. Attempting to replicate the interaction while addressing the original authors’ concerns regarding the protocol for the first replication study, we developed a new protocol in collaboration with the original authors. We used four university sites ( N = 754) chosen for similarity to the site of the original study to conduct a high-powered, preregistered replication focused primarily on the interaction effect. Results from these sites did not support the interaction or the main effect and were comparable to results obtained at six additional universities that were less similar to the original site. Post hoc analyses did not provide strong evidence for statistical inconsistency between the original study’s estimates and our estimates; that is, the original study’s results would not have been extremely unlikely in the estimated distribution of population effects in our sites. We also collected data from a new Mechanical Turk sample under the first replication study’s protocol, and results were not meaningfully different from those obtained with the new protocol at universities similar to the original site. Secondary analyses failed to support proposed substantive mechanisms for the failure to replicate.

Challenges and suggestions for defining replication “success” when effects may be heterogeneous: Comment on Hedges and Schauer (2019).

“…For ease of presentation, these reanalyses have been simplified somewhat compared with the models originally fit by Mathur et al (in press), but yielded qualitatively similar results.…”

“…Under Hedges and Schauer’s (2019) framework, a set of replication studies all with exactly null point estimates would be considered a perfect replication “success,” even if the original study hypothesized and supported a large positive effect. For example, we previously conducted a multisite replication (Mathur et al, in press) of an original study that reported a positive effect regarding a form of magical thinking (mean difference = 1.03; 95% CI [0.09, 1.97]; p = .03; Risen & Gilovich, 2008). 1 These 11 replications ( n = 4,441 total) showed little apparent heterogeneity ( Q = 1.85, p = 1.00), such that Hedges and Schauer’s (2019) test would fail to reject the null hypothesis of exact replication.…”