New statistical metrics for multisite replication projects

Abstract

Increasing interest in replicability in the social sciences has engendered novel designs for replication projects in which multiple sites replicate an original study. At least 134 such "many-to-one" replications have been completed since 2014 or are currently ongoing. These designs have unique potential to help estimate whether the original study is statistically consistent with the replications and to re-assess the strength of evidence for the scientific effect of interest. However, existing statistical analyses generally focus on single replications; when applied to many-to-one designs, they provide an incomplete view of aggregate evidence and can lead to unduly pessimistic conclusions about replication success. We therefore propose new statistical metrics representing: (1) the probability that the original study's estimated effect size would be as extreme or more extreme than it actually was, if in fact the original study is statistically consistent with the replications; (2) the proportion of true effects agreeing in direction with the original study. Generalized versions of the second metric allow consideration only of true effects of non-negligible size; they estimate the proportion of true effects of scientifically meaningful size in the same direction as the estimate of the original study and, secondly, the proportion of effects of meaningful size in the direction opposite the original study's estimate. These metrics apply under the assumptions that the true effects are normally distributed and that the replication point estimates are normally distributed around their corresponding true effects. We provide an R package ("Replicate").

A Brief Guide to Evaluate Replications

“…Under this situation, one could informally say that an effect is "replicable." When several replications are available, a more quantitative meta-analytic approach can be taken: an effect can be considered "replicable" when the meta-analytic ES estimate excludes zero and is consistent with the original ES point estimate (also replication scenario #1, see Panel A Figure 1; see also Mathur & VanderWeele, 2018).…”

Many Labs 5: Testing pre-data collection peer review as an intervention to increase replicability

“…First, we will conduct a form of equivalence testing by assessing, for each study, the proportion of effect sizes in the potentially heterogeneous distribution underlying the replications that are greater than an effect size of 0 and greater than a small effect size (e.g., r = .10; Mathur & VanderWeele, 2017). We will also examine the observed replicability rate (defined as the proportion of replications for each study that estimated a statistically significant point estimate in the same direction as that of the original study) compared to the rate we would expect in ideal circumstances (e.g., those without selective analyses or reporting) following the guidelines of Mathur and VanderWeele (2017). Finally, we will examine the consistency of each set of replications (either separated by protocol or pooled) with the results of the original studies and RP:P replications, again following the guidelines of Mathur and VanderWeele (2017).…”

“…We will also examine the observed replicability rate (defined as the proportion of replications for each study that estimated a statistically significant point estimate in the same direction as that of the original study) compared to the rate we would expect in ideal circumstances (e.g., those without selective analyses or reporting) following the guidelines of Mathur and VanderWeele (2017). Finally, we will examine the consistency of each set of replications (either separated by protocol or pooled) with the results of the original studies and RP:P replications, again following the guidelines of Mathur and VanderWeele (2017). ]…”

Discussion on the Meeting on ‘Signs and Sizes:Understanding and Replicating Statistical Findings’

“…Other considerations are also relevant to replication success. One can assess the extent to which the original study is statistically consistent with the replication study, in that its data are compatible with the same underlying population parameter that is unbiasedly estimated in the replications (Mathur and VanderWeele, ). This method assesses whether the replication and original studies’ point estimates are more distant than expected by chance, accounting for statistical uncertainty in both, and hence are conceptually different from Held's reverse Bayesian sensitivity analysis on the original study.…”

“…This method assesses whether the replication and original studies’ point estimates are more distant than expected by chance, accounting for statistical uncertainty in both, and hence are conceptually different from Held's reverse Bayesian sensitivity analysis on the original study. For example, the consistency metric accommodates ‘non‐significant’ original studies and those with a priori plausible or implausible findings; it also applies readily to multisite replication studies, accounting for effect heterogeneity (Mathur and VanderWeele, ). With multisite replications, one can also reassess the strength of evidence for the effect under investigation by meta‐analytic methods to estimate the proportion of true effect sizes in the replications that are stronger than a chosen threshold of scientific importance (Mathur and VanderWeele, ).…”