Estimating the deep replicability of scientific findings using human and artificial intelligence

Abstract

Replicability tests of scientific papers show that the majority of papers fail replication. Moreover, failed papers circulate through the literature as quickly as replicating papers. This dynamic weakens the literature, raises research costs, and demonstrates the need for new approaches for estimating a study’s replicability. Here, we trained an artificial intelligence model to estimate a paper’s replicability using ground truth data on studies that had passed or failed manual replication tests, and then tested the model’s generalizability on an extensive set of out-of-sample studies. The model predicts replicability better than the base rate of reviewers and comparably as well as prediction markets, the best present-day method for predicting replicability. In out-of-sample tests on manually replicated papers from diverse disciplines and methods, the model had strong accuracy levels of 0.65 to 0.78. Exploring the reasons behind the model’s predictions, we found no evidence for bias based on topics, journals, disciplines, base rates of failure, persuasion words, or novelty words like “remarkable” or “unexpected.” We did find that the model’s accuracy is higher when trained on a paper’s text rather than its reported statistics and that n-grams, higher order word combinations that humans have difficulty processing, correlate with replication. We discuss how combining human and machine intelligence can raise confidence in research, provide research self-assessment techniques, and create methods that are scalable and efficient enough to review the ever-growing numbers of publications—a task that entails extensive human resources to accomplish with prediction markets and manual replication alone.

Replicability, Robustness, and Reproducibility in Psychological Science

“…The results suggest that statistical properties like sample sizes, p-values and effect sizes of the original studies, and whether the effects are main effects or interaction effects are predictive of successful replication (Altmejd et al, 2019). Models trained on the original papers' narrative text performed better than those on reported statistics (Yang et al, 2020). In both studies, the models perform similarly to the prediction markets on the same data.…”

“…The results suggest that statistical properties like sample sizes, p-values, and effect sizes of the original studies as well as whether the effects are main effects or interaction effects are predictive of successful replication. Yang et al (2020) also predict replicability with machine learning models, starting with the Open Science Collaboration (2015) as training data, and doing out-of-sample tests on a more extensive set of replications from psychology as well as economics compared to Altmejd et al They compare predictive models trained on either the original papers' narrative (text), reported statistics or both narrative and reported statistics. The model's accuracy is higher when trained on narrative than when trained on reported statistics, and the results also suggest that higher word combinations correlate with replication.…”

Predicting replicability—Analysis of survey and prediction market data from large-scale forecasting projects

“…The p-values of the findings has been shown to be correlated with the replication outcomes [ 20 , 21 ]. In particular two other replication based forecasting attempts has shown that p-values are informative to a machine learning algorithm [ 33 , 34 ]. We here test this relationship using the pooled market data.…”

Predicting reliability through structured expert elicitation with repliCATS (Collaborative Assessments for Trustworthy Science)

“…In the latter, participants trade contracts that give a small payoff if and only if the study is replicated and prices are used to derive the likelihood of replicability. Some other studies have fully automated predictions of replicability, such as machine learning techniques [9,10] although such techniques are not considered further in this paper.…”