Rewarding Replications

Abstract

Although replications are vital to scientific progress, psychologists rarely engage in systematic replication efforts. In this article, we consider psychologists' narrative approach to scientific publications as an underlying reason for this neglect and propose an incentive structure for replications within psychology. First, researchers need accessible outlets for publishing replications. To accomplish this, psychology journals could publish replication reports in files that are electronically linked to reports of the original research. Second, replications should get cited. This can be achieved by cociting replications along with original research reports. Third, replications should become a valued collaborative effort. This can be realized by incorporating replications in teaching programs and by stimulating adversarial collaborations. The proposed incentive structure for replications can be developed in a relatively simple and cost-effective manner. By promoting replications, this incentive structure may greatly enhance the dependability of psychology's knowledge base.

Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field

“…Inviting the initial author to review replication studies can help reduce this, and (in the case of a Registered Report) the initial authors can be invited to provide a Stage 1 review before data collection (see Marsden, Morgan-Short, Trofimovich, & Ellis, 2018). Even more transparent practices that may promote more and higher quality replication, reduce publication bias, and reduce perceptions of bullying include (a) publishing open reviews and authors' responses to reviews (e.g., in BMC Psychology; Laws, 2016); (b) giving initial authors an automatic right to a peer-reviewed published commentary (e.g., in Perspectives in Psychological Science; in our sample, we found one such example, Kanno, 2000); and (c) adversarial collaborations (Coyne, 2016;Kahneman, 2014;Koole & Lakens, 2012;Mellers, Hertwig, & Kahneman, 2001), where researchers who account for phenomena differently agree to work together following a single protocol. We raise awareness of the existence of these more extreme measures but hope that the other mechanisms that we recommend, such as transparent materials and data and the reviewing of methods prior to data collection, serve to reduce any perception of bullying that independent replication may engender.…”

“…In psychology, Makel et al (2012) found 91.7% supported initial findings when there was author overlap, versus 64.6% when there was no overlap. Given the high rate of reproducibility with author overlap, Koole and Lakens (2012) focused only on independent replications in their set of recommendations for replication, arguing that "the most compelling direct replications are conducted independently by different researchers than the original study" (p. 609). This was a key motivator for the preregistered multisite replications published by and Practices in Psychological Science), in which research teams all have access to the same materials but conduct the study independently (and in some cases, do not look at the data until they have been passed to the replication convener or coordinating editor).…”

“…The mean time between a study and its published replication was 6.64 years (SD = 6.16, Mdn = 5, mode = 1, k = 11, range = 0-37). This time delay demonstrates the need for a sustained infrastructure to help replications to be performed and published more quickly because ascertaining the generalizability and reliability of study findings can reduce the chance of self-perpetuating misinformed agendas and of drawing implications for practice too hastily (see also Koole & Lakens, 2012).…”

Reproducibility: A Primer on Semantics and Implications for Research

“…3 Scientists across numerous disciplines have expressed serious concerns and engaged in both philosophical and practical discussions about reproducible research. Disciplines represented in these discussions include biology, 4 biomedical 5 and preclinical research, 6,7 business and organizational studies, [8][9][10][11][12] computational sciences, 13,14 drug discovery, 15 economics, 16,17 education, [18][19][20][21][22] epidemiology and statistics, [23][24][25] genetics, 26 immunology, 27 policy research, 28 political science, [29][30][31][32][33][34][35][36] psychology, 29,[37][38][39][40][41][42][43] and sociology. 44 As a case in point: research indicates that more than half of psychology studies fail reproducibility tests.…”

Modification of episodic memories by novel learning: a failed replication study

“…This emphasizes the importance of independent replications; an appeal that has been made repeatedly in recent years to ensure the self-correcting nature of psychological science (e.g. Asendorpf et al, 2013; Koole & Lakens, 2012; Makel, Plucker, & Hegarty, 2012; Nosek, Spies, & Motyl, 2012; Pashler & Wagenmakers, 2012). In this spirit, we attempted to replicate Wichert et al’s (experiment 1, 2013a) findings using similar procedures, manipulation, measures, and population.…”