Replicator degrees of freedom allow publication of misleading failures to replicate

Abstract

In recent years, the field of psychology has begun to conduct replication tests on a large scale. Here, we show that “replicator degrees of freedom” make it far too easy to obtain and publish false-negative replication results, even while appearing to adhere to strict methodological standards. Specifically, using data from an ongoing debate, we show that commonly exercised flexibility at the experimental design and data analysis stages of replication testing can make it appear that a finding was not replicated when, in fact, it was. The debate that we focus on is representative, on key dimensions, of a large number of other replication tests in psychology that have been published in recent years, suggesting that the lessons of this analysis may be far reaching. The problems with current practice in replication science that we uncover here are particularly worrisome because they are not adequately addressed by the field’s standard remedies, including preregistration. Implications for how the field could develop more effective methodological standards for replication are discussed.

Machine Learning in Psychometrics and Psychological Research

“…The author (Cumming, 2008) proved, in a simulation study of 25 repetitions, that a result in the first experiment significant at p < 0.05 in the replications may vary from p < 0.001 to p = 0.76 (with a 10% chance of p > 0.44) showing that p is a very unreliable measure. To complicate the landscape, some researchers have also highlighted how failed replication are not immune from the same type of error that may be detected in the original studies (Bressan, 2019) and false negatives in replication studies have recently attracted attention (Bryan et al, 2019).…”

Les pratiques de recherche ouvertes en psychologie

“…Sur 100 tentatives de réplications de recherches en psychologie, seules 36 s’avèrent « réussies ». Il faut cependant bien garder à l’esprit que la simple attribution du qualificatif « échec » ou « réussite » lorsque l’on parle d’une réplication fait l’objet de très nombreux débats, amenant naturellement à des constats différents suivant les critères retenus, sans même évoquer la question complexe de la détermination des véritables causes de l’échec d’une réplication ( Bryan, Yeager, & O’Brien, 2019 ; Gilbert, King, Pettigrew, & Wilson, 2016 ; Nelson, Simmons, & Simonsohn, 2018 ; Stanley, Carter, & Doucouliagos, 2018 ; Zwaan, Etz et al, 2018 ).…”

Replication and the Establishment of Scientific Truth

“…There are numerous examples in the literature of methodological liberties replication researchers have taken in attempts to disprove original findings (Doyen et al, 2012;Gerber et al, 2016). The "replicator degrees of freedom" have been shown to lead to unwarranted claims of replication failures (Ramscar, 2016;Bryan et al, 2019). Thus, it cannot be assumed that so-called independent replications are unbiased.…”

“…Thus, it cannot be assumed that so-called independent replications are unbiased. In fact, it has been suggested that there is an incentive to find the null result refuting the original finding (Bryan et al, 2019). Refutations can readily be obtained by liberal uses of replicator degrees of freedom [e.g., prior selection of experimental designs and analyses (Bryan et al, 2019;Sherman and Rivers, in press)].…”

“…In fact, it has been suggested that there is an incentive to find the null result refuting the original finding (Bryan et al, 2019). Refutations can readily be obtained by liberal uses of replicator degrees of freedom [e.g., prior selection of experimental designs and analyses (Bryan et al, 2019;Sherman and Rivers, in press)]. Furthermore, people's general sensitivity to social influences (Bavel et al, 2016) makes it relatively easy to conduct, unwittingly or not, replication experiments to produce refuting evidence.…”