Response to Comment on “Estimating the reproducibility of psychological science”

Abstract

Gilbert et al. conclude that evidence from the Open Science Collaboration's Reproducibility Project: Psychology indicates high reproducibility, given the study methodology. Their very optimistic assessment is limited by statistical misconceptions and by causal inferences from selectively interpreted, correlational data. Using the Reproducibility Project: Psychology data, both optimistic and pessimistic conclusions about reproducibility are possible, and neither are yet warranted.A cross multiple indicators of reproducibility, the Open Science Collaboration (1) (OSC2015) observed that the original result was replicated in~40 of 100 studies sampled from three journals. Gilbert et al. (2) conclude that the reproducibility rate is, in fact, as high as could be expected, given the study methodology. We agree with them that both methodological differences between original and replication studies and statistical power affect reproducibility, but their very optimistic assessment is based on statistical misconceptions and selective interpretation of correlational data.Gilbert et al. focused on a variation of one of OSC2015's five measures of reproducibility: how often the confidence interval (CI) of the original study contains the effect size estimate of the replication study. They misstated that the expected replication rate assuming only sampling error is 95%, which is true only if both studies estimate the same population effect size and the replication has infinite sample size (3, 4). OSC2015 replications did not have infinite sample size. In fact, the expected replication rate was 78.5% using OSC2015's CI measure (see OSC2015's supplementary information, pp. 56 and 76; https://osf.io/k9rnd). By this measure, the actual replication rate was only 47.4%, suggesting the influence of factors other than sampling error alone.Within another large replication study, "Many Labs" (5) (ML2014), Gilbert et al. found that 65.5% of ML2014 studies would be within the CIs of other ML2014 studies of the same phenomenon and concluded that this reflects the maximum reproducibility rate for OSC2015. Their analysis using ML2014 is misleading and does not apply to estimating reproducibility with OSC2015's data for a number of reasons.First, Gilbert et al.'s estimates are based on pairwise comparisons between all of the replications within ML2014. As such, for roughly half of their failures to replicate, "replications" had larger effect sizes than "original studies," whereas just 5% of OSC2015 replications had replication CIs exceeding the original study effect sizes.Second, Gilbert et al. apply the by-site variability in ML2014 to OSC2015's findings, thereby arriving at higher estimates of reproducibility. However, ML2014's primary finding was that by-site variability was highest for the largest (replicable) effects and lowest for the smallest (nonreplicable) effects. If ML2014's primary finding is generalizable, then Gilbert et al.'s analysis may leverage by-site variability in ML2014's larger effects to exaggerate the effect of by-sit...

Contextual sensitivity in scientific reproducibility

“…However, some people have speculated that the original authors preregistered concern only because they were aware that their studies were relatively weak based on other factors affecting replication (e.g., small effect sizes, underpowered designs) (59). The OSC authors also argued that authors who were less confident of their study's robustness may have been less likely to endorse the replications (47).…”

“…For instance, a recent critique of the Reproducibility Project alleged that several replication studies differed significantly from the original studies, undercutting any inferences about lack of reproducibility in psychology (56). The allegation that low-fidelity replication attempts undercut the validity of the Reproducibility Project launched a debate about the role of contextual factors in several replication failures, both in print (47) and in subsequent online commentaries (e.g., refs. 57-59).…”

“…suggested, "Once we realize that effects are contextually bound, a next step is to study how they vary" (33). Indeed, the OSC authors correctly note that "there is no such thing as exact replication" in the field of psychology (47). Although the ideal methods section should include enough detail to permit a direct replication, this seemingly reasonable demand is rarely satisfied in psychology, because human behavior is easily affected by seemingly irrelevant factors (48).…”

Too Good to be False: Nonsignificant Results Revisited

“…This was also noted by both the original RPP team (Open Science Collaboration, 2015;Anderson, 2016) and in a critique of the RPP (Gilbert, King, Pettigrew, & Wilson, 2016). Replication efforts such as the RPP or the Many Labs project remove publication bias and result in a less biased assessment of the true effect size.…”

“…Although these studies suggest substantial evidence of false positives in these fields, replications show considerable variability in resulting effect size estimates (Klein, et al, 2014;Stanley, & Spence, 2014). Therefore caution is warranted when wishing to draw conclusions on the presence of an effect in individual studies (original or replication;Open Science Collaboration, 2015;Gilbert, King, Pettigrew, & Wilson, 2016;Anderson, et al 2016).…”