Why Replication Is Overrated

Abstract

Why Replication is Overrated Current debates about the replication crisis in psychology take it for granted that direct replication is valuable and focus their attention on questionable research practices in regard to statistical analyses. This paper takes a broader look at the notion of replication as such. It is argued that all experimentation/replication involves individuation judgments and that research in experimental psychology frequently turns on probing the adequacy of such judgments. In this vein, I highlight the ubiquity of conceptual and material questions in research, and I argue that replication is not as central to psychological research as it is sometimes taken to be. 1. Introduction: The "Replication Crisis" In the current debate about replicability in psychology, we can distinguish between (1) the question of why not more replication studies are done (e.g., Romero 2017) and (2) the question of why a significant portion (more than 60%) of studies, when they are done, fail to replicate (I take this number from the Open Science Collaboration, 2015). Debates about these questions have been dominated by two assumptions, namely, first, that it is in general desirable that scientists conduct replication studies that come as close as possible to the original, and second, that the low replication rate can often be attributed to statistical problems with many initial studies, sometimes referred to as "p-hacking" and "data-massaging." 1 1 An important player in this regard is the statistician Andrew Gelman who has been using his blog as a public platform to debate methodological problems with mainstream social psychology (http://andrewgelman.com/).

Replication crisis = trust crisis? The effect of successful vs failed replications on laypeople’s trust in researchers and research

“…altering hypotheses in the light of knowing the results). On the other hand, a failed replication could be the result different underlying “conceptual scope,” namely whether researchers put very narrow assumptions to the test in (direct) replication (such as, “can this specific effect be replicated using the exact same methods and materials”), or test broader assumptions about the scope and nature of an effect found previously (Feest, 2019). It is an interesting question whether evaluations about replications would be facilitated when participants learn about (a) a series of replicated or non-replicated studies (e.g.…”

Towards a Contextual Approach to Data Quality

“…As a consequence, what reproducibility means, what can and should be reproduced and the degree to which reproducibility is possible-if at all-change significantly in connection to the specific context of research. With a focus of experimental practices, Uljana Feest has argued that the role of reproducibility is not particularly central in this context [41] and Sabina Leonelli has claimed that reproducibility requirements should be tailored to the contextual features, circumstances and goals of a specific scientific project or area of research, as well as the assumptions, values and judgements that are involved in practice [42]. This direction in the literature on the reproducibility crisis further intersects with the contextual approach I have presented.…”

Replicability, Robustness, and Reproducibility in Psychological Science

“…Replicability does not guarantee validity nor that the knowledge is applicable. Theorists vary in their weighting of which areas are necessary to improve to advance knowledge (Devezer et al, 2019;Feest, 2019;Frank et al, 2017;Leonelli, 2018). And, at present, there is little empirical evidence to advance these debates.…”