Predicting the Replicability of Social Science Lab Experiments

Abstract

We measure how accurately replication of experimental results can be predicted by a black-box statistical model. With data from four large- scale replication projects in experimental psychology and economics, and techniques from machine learning, we train a predictive model and study which variables drive predictable replication.The model predicts binary replication with a cross validated accuracy rate of 70% (AUC of 0.79) and relative effect size with a Spearman ρ of 0.38. The accuracy level is similar to the market-aggregated beliefs of peer scientists (Camerer et al., 2016; Dreber et al., 2015). The predictive power is validated in a pre-registered out of sample test of the outcome of Camerer et al. (2018b), where 71% (AUC of 0.73) of replications are predicted correctly and effect size correlations amount to ρ = 0.25.Basic features such as the sample and effect sizes in original papers, and whether reported effects are single-variable main effects or two- variable interactions, are predictive of successful replication. The models presented in this paper are simple tools to produce cheap, prognostic replicability metrics. These models could be useful in institutionalizing the process of evaluation of new findings and guiding resources to those direct replications that are likely to be most informative.

Nonreplicable publications are cited more than replicable ones

“…Yet, the relationship between replicability and citations is not significantly different across the three replication projects. When we include several individual characteristics of the studies replicated [based on (15)], such as the number of authors, the rate of male authors, and the characteristics of the experiment (location, language, and online implementation), as well as the field in which the paper was published (16), the relationship between replicability and citations is qualitatively unchanged (the same occurs if we control for the highest seniority level among the authors). In the Supplementary Materials, we provide a robustness analysis based on specification curves that display the robustness of results across 36 different regression models (17).…”

Research Replication Prediction Using Weakly Supervised Learning

“…Machine learning serves as a much more efficient method to conduct replication prediction. Altmejd et al (2019) applied ML methods on the data from four large-scale replication projects in experimental psychology and economics and studied which variables drive predictable replication. But they used only statistics features such as p-value, sample size, etc.…”

“…To include as many annotated data points as possible, we adopt the most basic binary model that defines replication success as a "statistically significant (p-value <= 0.05) effect in the same direction as in the original study." (Altmejd et al, 2019) The Among 399 annotated samples, 201 samples are labeled as '1' (replicable). The remaining 198 samples are annotated as '0' (non-replicable).…”

“…Recently, efforts have been noted to use machine learning as a much cheaper and more efficient alternative to provide an informative replication prediction Yang, 2018;Altmejd et al, 2019). It has been reported that with simple machine learning models, a predicted accuracy of 71% can be achieved.…”

Doing Science: Methods and Philosophy

“…2 Objectivity can be attributed, among others, to scientific measurements, tools for development/improvement of scientific theories, and/or to true-to-nature explanations. It ensures that study outcomes are not biased (e.g., over estimation of drug efficacy, under estimation of risk; Goldacre, 2014), positive research results are not false-positives (to a larger proportion than is allowed by the statistical method; Simmons et al, 2011), and are independently reproducible by other scientists (Altmejd et al, 2019;Lindsay, 2015;Simons, 2014; van Bavel, Mende-Siedlecki, Brady, & Reinero, 2016). Dr. Summers considers objectivity to be essential to science 3 and its absence to be a cause of the crisis that threatens the foundations of her research field.…”