Bayesian tests to quantify the result of a replication attempt.

Abstract

Replication attempts are essential to the empirical sciences. Successful replication attempts increase researchers' confidence in the presence of an effect, whereas failed replication attempts induce skepticism and doubt. However, it is often unclear to what extent a replication attempt results in success or failure. To quantify replication outcomes we propose a novel Bayesian replication test that compares the adequacy of two competing hypotheses. The first hypothesis is that of the skeptic and holds that the effect is spurious; this is the null hypothesis that postulates a zero effect size, H 0 : δ = 0. The second hypothesis is that of the proponent and holds that the effect is consistent with the one found in the original study, an effect that can be quantified by a posterior distribution. Hence, the second hypothesis -the replication hypothesis-is given by H r : δ ∼ "posterior distribution from original study". The weighted likelihood ratio between H 0 and H r quantifies the evidence that the data provide for replication success and failure. In addition to the new test, we present several other Bayesian tests that address different but related questions concerning a replication study. These tests pertain to the independent conclusions of the separate experiments, the difference in effect size between the original experiment and the replication attempt, and the overall conclusion based on the pooled results. Together, this suite of Bayesian tests allows a relatively complete formalization of the way in which the result of a replication attempt alters our knowledge of the phenomenon at hand. The use of all Bayesian replication tests is illustrated with three examples from the literature. For experiments analyzed using the t test, computation of the new replication test only requires the t values and the numbers of participants from the original study and the replication study.

A Bayesian approach to mitigation of publication bias

“…For example, for the unpaired t test, η true = ϕδ true with ϕ = n 1 +n 2 n 1 n 2 . Transformations for other inferential tests are well known (e.g., Ellis, 2010; or succinctly in Footnote 4 of Verhagen & Wagenmakers, 2014). To define a prior for the TS, we propose a unit information prior on the ES η true : η true ∼ N (0, 1).…”

Making Replication Mainstream

“…A second approach is the "replication Bayes factor" approach (Ly, Etz, Marsman, Wagenmakers, 2017;Verhagen & Wagenmakers, 2014;.…”

A reminder on millisecond timing accuracy and potential replication failure in computer-based psychology experiments: An open letter

“…Areas for concern range from experimenter expectancy and statistical power through to publication bias and the file drawer problem to outright research fraud. Some have gone so far as to suggest that Bayesian tests might be applied to quantify the results or efficacy of replication attempts so that the field might know which studies are more valid and go some way to ameliorate the issue (Verhagen & Wagenmakers, 2014). However, few have considered that a growing proportion of research in modern psychology is conducted using a computer, whether that be under 'controlled conditions' in a laboratory or more widely online across the web.…”