Internal conceptual replications do not increase independent replication success

Abstract

Recently, many psychological effects have been surprisingly difficult to reproduce. This article asks why, and investigates whether conceptually replicating an effect in the original publication is related to the success of independent, direct replications. Two prominent accounts of low reproducibility make different predictions in this respect. One account suggests that psychological phenomena are dependent on unknown contexts that are not reproduced in independent replication attempts. By this account, internal replications indicate that a finding is more robust and, thus, that it is easier to independently replicate it. An alternative account suggests that researchers employ questionable research practices (QRPs), which increase false positive rates. By this account, the success of internal replications may just be the result of QRPs and, thus, internal replications are not predictive of independent replication success. The data of a large reproducibility project support the QRP account: replicating an effect in the original publication is not related to independent replication success. Additional analyses reveal that internally replicated and internally unreplicated effects are not very different in terms of variables associated with replication success. Moreover, social psychological effects in particular appear to lack any benefit from internal replications. Overall, these results indicate that, in this dataset at least, the influence of QRPs is at the heart of failures to replicate psychological findings, especially in social psychology. Variable, unknown contexts appear to play only a relatively minor role. I recommend practical solutions for how QRPs can be avoided.Electronic supplementary materialThe online version of this article (doi:10.3758/s13423-016-1030-9) contains supplementary material, which is available to authorized users.

Merging Structural and Process–Related Approaches to the Study of Agreeableness: A Preregistered Replication and Extension Attempt

“…It is also worth noting that both behavioural tasks used in the current study had been supported by conceptual replications in the original research articles. Our results are in line with other work that has found that conceptual replications do not necessarily increase the chances of independent, direct replication success (Kunert, ).…”

Merging Structural and Process-related Approaches to the Study of Agreeableness: A Preregistered Replication and Extension Attempt

“…It is also worth noting that both behavioural tasks used in the current study had been supported by conceptual replications in the original research articles. Our results are in line with other work that has found that conceptual replications do not necessarily increase the chances of independent, direct replication success (Kunert, 2016).…”

Low replicability can support robust and efficient science

“…On this account, what matters is not whether an exact replication can reproduce the original effect but whether the underlying theory finds further support in conceptual replications (studies that use a variety of different manipulations or measures to operationalize the crucial theoretical variables) 6 . Contrary to this position, the success of independent replications is no greater for effects that were initially reported together with conceptual replications than effects that were reported in isolation 7 .…”

“…a Asterisks indicate significant departure (maximum significant p < :0007, two-tailed single-sample t-test) from midpoint of scale (7)…”