Observe, hypothesize, test, repeat: Luttrell, Petty and Xu (2017) demonstrate good science

Abstract

Many Labs 3 (Ebersole et al., 2016) failed to replicate a classic finding from the Elaboration Likelihood Model of persuasion (Cacioppo, Petty, & Morris, 1983; Study 1). Petty and Cacioppo (2016) noted possible limitations of the Many Labs 3 replication (Ebersole et al., 2016) based on the cumulative literature. Luttrell, Petty, and Xu (2017) subjected some of those possible limitations to empirical test. They observed that a revised protocol obtained evidence consistent with the original finding that the Many Labs 3 protocol did not. This observe-hypothesize-test sequence is a model for scientific inquiry and critique. To test whether these results advance replicability and knowledge transfer, we conducted direct replications of Luttrell et al. in nine locations (Total N = 1,219). We successfully replicated the interaction of need for cognition and argument quality on persuasion using Luttrell et al.'s optimal design (albeit with a much smaller effect size; p < .001; f 2 = .025, 95%CI [.006, .056]) but failed to replicate the interaction that indicated that Luttrell et al.'s optimal protocol performed better than the Many Labs 3 protocol (p = .135, pseudo R 2 = .002). Nevertheless, pragmatically, we favor the Luttrell et al. protocol with large samples for future research using this paradigm.

A Validity-Based Framework for Understanding Replication in Psychology

“…Luttrell et al (2017) successfully replicated the Cacioppo et al (1983) NC Â AQ interaction when using the optimal procedure and also replicated the failure to find an effect when using the Ebersole et al (2016) protocol. Then in another replication literature first, these findings were further supported by an independent multi-lab replication of Luttrell et al (2017) conducted by Ebersole et al (2017). They too failed to find a significant NC Â AQ interaction effect with the Ebersole et al (2016) protocol (replicating their and Luttrell et al's replication failure with those materials), but they did obtain a significant interaction effect with the Luttrell et al (2017) protocol.…”

“…Is this particular replication study a rare outlier, or a comparatively typical representative of the larger replication literature, or does it fall somewhere in between? It is impossible to answer this question because there are so few studies in which critics of replication efforts attempt to empirically validate their speculations, and to date there is just one example in which the replicators attempted to validate the insights of the critics of their replication effort (Ebersole et al, 2017). However, knowing whether critics of replication efforts are correct in their speculations about why a replication effort failed or not could provide valuable insights into the broader implications of disappointing replication rates.…”

Making sense of replications

“…It is possible, for example, that differences in the methodologies that were thought to be irrelevant are actually important (Hines et al, 2014). Indeed, a failed replication can lead to a better understanding of a phenomenon if it results in the generation of new hypotheses to explain how the original and replication methodologies produced different results and, critically, leads to follow-up experiments to test these hypotheses (Ebersole et al, 2017). …”

The Role of Replication Studies in Theory Building

“…Fo e a ple, it has ee ell e og ised that epli atio atte pts should ideally inform theory in an iterative way, with theory informing replication design, replication results informing theory, and on again (e.g. Earp & Trafimow, 2015;Ebersole et al, 2017;Klein et al, 2014a). However these points are (unsurprisingly) rarely seen as undermining the aim and practice of replication in general.…”