A New Replication Norm for Psychology

Abstract

There is growing consensus that psychology has a replicability "crisis of confidence" [70,69,89,66,67], stemming from the fact that a growing number of findings cannot be replicated via high-powered independent replication attempts that duplicate the original methodology as closely as possible. Across all areas of psychology, there is a growing list of (prominent) findings that have not held up to independent replication attempts, including findings from cognitive psychology (retrieval- [38,17,51,18,31,57,72,71,94,44,46,52,34]).More generalizable evidence supporting a general replicability problem comes from an ambitious and unprecedented large-scale crowdsourced project, the Reproducibility Project [66,67]. In this project, researchers were unable to replicate about 60% (out of 100) of findings from the 2008 issues of Psychological Science, Journal of Personality and Social Psychology, and Journal of Experimental Psychology: Learning, Memory, and Cognition [68]. In another large-scale meta-scientific investigation, about 70% (16 out of 23) of important findings from cognitive and social psychology could also not be replicated [64]. Though there are different ways to interpret successful versus unsuccessful replication results [91], taken together, these observations strongly suggest psychology currently has a general replicability problem (as do several other areas of science including cancer cell biology and cardiovascular health literatures [8,75]). New Initiatives and ReformsSeveral new initiatives have been launched to improve research practices in order to increase the reliability of findings in psychology. For instance, higher reporting standards have recently been instituted at several prominent psychology journals [21,87,88,47]. At such journals (e.g., Psychological Science, Memory & Cognition, Attention, ORIGINAL RESEARCH REPORT A New Replication Norm for PsychologyEtienne P. LeBel * In recent years, there has been a growing concern regarding the replicability of findings in psychology, including a mounting number of prominent findings that have failed to replicate via high-powered independent replication attempts. In the face of this replicability "crisis of confidence", several initiatives have been implemented to increase the reliability of empirical findings. In the current article, I propose a new replication norm that aims to further boost the dependability of findings in psychology. Paralleling the extant social norm that researchers should peer review about three times as many articles that they themselves publish per year, the new replication norm states that researchers should aim to independently replicate important findings in their own research areas in proportion to the number of original studies they themselves publish per year (e.g., a 4:1 original-to-replication studies ratio). I argue this simple approach could significantly advance our science by increasing the reliability and cumulative nature of our empirical knowledge base, accelerating our theoretical understanding of psycholo...

Meta-analysis to integrate effect sizes within an article: Possible misuse and Type I error inflation.

“…Again, we need to emphasize that the issue being discussed here does not lie in (internal) meta-analysis per se, but in researchers' (implicit) motivation to use the strategic flexible stopping rule to maximize positive results with NHST (see also, Ioannidis, 2005;Klein et al, 2014;Open Science Collaboration, 2012, 2013, 2015Pashler & Harris, 2012;Pashler & Wagenmakers, 2012). Without using the flexible stopping rule, one can reliably keep false positive error rates to the conventional nominal level (e.g., 5%) regardless of how many studies are integrated within an article.…”

“…In fact, a recent large-scale reproducibility project (Open Science Collaboration, 2015) found that the effect sizes in (high-powered) replication studies were half the magnitude of the original effects of 100 articles in high-profile journals from cognitive and social psychology (see also, Klein et al, 2014; Open Science Collaboration, 2012, 2013; Pashler & Harris, 2012; Pashler & Wagenmakers, 2012). LeBel (2015) also argued that this issue is not limited to cognitive and social psychology, but is prevalent in most fields of psychology (or perhaps in other scientific fields, e.g., neuroscience). These findings clearly demonstrate the value of conducting high powered research in psychology to accurately ascertain effect size.…”

Adjusting Sample Sizes for Different Categories of Embodied Cognition Research

“…In response to the current replication crisis in psychology (for discussions, see Pashler and Wagenmakers, 2012; Maxwell et al, 2015), several solutions have been proposed to improve the quality of psychological research (e.g., Chambers, 2013; Simons, 2014; LeBel, 2015; for overviews, see Ferguson, 2015; Zwaan et al, 2017). Benjamin et al (2018) argue for a change of the standard 0.05 alpha level and instead support to lower the default alpha value for novel findings in the field of psychology to 0.005.…”

Improving the Replicability of Psychological Science Through Pedagogy

“…One potential solution to this problem is to make replication an explicit part of pedagogy: that is, to teach students about experimental methods by asking them to run replication studies (Frank & Saxe, 2012; Grahe et al, 2012). Despite enthusiasm for this idea (Everett & Earp, 2015; King et al, 2016; LeBel, 2015; Standing, 2016), there is limited data beyond anecdotal reports and individual projects (e.g., Lakens, 2013; Phillips et al, 2015) to suggest that the pedagogical use of replications should be adopted on a broader scale.…”