Expectations for Replications

Abstract

Failures to replicate published psychological research findings have contributed to a "crisis of confidence." Several reasons for these failures have been proposed, the most notable being questionable research practices and data fraud. We examine replication from a different perspective and illustrate that current intuitive expectations for replication are unreasonable. We used computer simulations to create thousands of ideal replications, with the same participants, wherein the only difference across replications was random measurement error. In the first set of simulations, study results differed substantially across replications as a result of measurement error alone. This raises questions about how researchers should interpret failed replication attempts, given the large impact that even modest amounts of measurement error can have on observed associations. In the second set of simulations, we illustrated the difficulties that researchers face when trying to interpret and replicate a published finding. We also assessed the relative importance of both sampling error and measurement error in producing variability in replications. Conventionally, replication attempts are viewed through the lens of verifying or falsifying published findings. We suggest that this is a flawed perspective and that researchers should adjust their expectations concerning replications and shift to a meta-analytic mind-set.

Non-cognate translation priming effects in the same–different task: evidence for the impact of “higher level” information

“…Currently, there is considerable discussion in the literature about the (lack of) replicability of a number of results (e.g., Pashler & Wagenmakers, 2012;Peng, 2011;Stanley & Spence, 2014). Replicating the priming effects observed in Experiment 4 would help to alleviate those types of concerns in the present situation.…”

Generalization versus contextualization in automatic evaluation revisited: A meta-analysis of successful and failed replications.

“…In the worst case, our initial findings may turn out as false positives if a comprehensive analysis of all studies fails to obtain statistically significant results. Of course, a certain proportion of replication failures have to be expected, because failed replications can also be due to unknown moderators and naturally occurring variations in effect sizes as a result of measurement and sampling error (Cummings, 2012;LeBel & Paunonen, 2011;Stanley & Spence, 2014). Nevertheless, the mere existence of several failed replications requires a thorough reassessment of the available evidence to reevaluate the validity of our theory.…”

Towards a Logical Framework for Latent Variable Modelling

“…… Thirty-six percent of replications had statistically significant results" (Open Science Collaboration, 2015, p. 4716-1). Subsequent analysis of the replication intention and efforts have suggested logical concerns to do with the expectations regarding replications themselves (Schmidt, 2010;Stanley & Spence, 2014). The logical framework described here may assist in answering logical questions for each project, such as ones regarding the nature of the psychological construct and its relation to psychological phenomena; or systematic aspects of measurement error which are otherwise not able to be tracked in either original studies, or attempted replications.…”

“…The logical framework described here may assist in answering logical questions for each project, such as ones regarding the nature of the psychological construct and its relation to psychological phenomena; or systematic aspects of measurement error which are otherwise not able to be tracked in either original studies, or attempted replications. This assessment may be used to feedback information regarding measurement error and/or construct to phenomenon relations, as raised in Schmidt (2010) and Stanley and Spence (2014).…”