Predicting replicability—Analysis of survey and prediction market data from large-scale forecasting projects

Abstract

The reproducibility of published research has become an important topic in science policy. A number of large-scale replication projects have been conducted to gauge the overall reproducibility in specific academic fields. Here, we present an analysis of data from four studies which sought to forecast the outcomes of replication projects in the social and behavioural sciences, using human experts who participated in prediction markets and answered surveys. Because the number of findings replicated and predicted in each individual study was small, pooling the data offers an opportunity to evaluate hypotheses regarding the performance of prediction markets and surveys at a higher power. In total, peer beliefs were elicited for the replication outcomes of 103 published findings. We find there is information within the scientific community about the replicability of scientific findings, and that both surveys and prediction markets can be used to elicit and aggregate this information. Our results show prediction markets can determine the outcomes of direct replications with 73% accuracy (n = 103). Both the prediction market prices, and the average survey responses are correlated with outcomes (0.581 and 0.564 respectively, both p < .001). We also found a significant relationship between p-values of the original findings and replication outcomes. The dataset is made available through the R package “pooledmaRket” and can be used to further study community beliefs towards replications outcomes as elicited in the surveys and prediction markets.

Using prediction markets to predict the outcomes in the Defense Advanced Research Projects Agency's next-generation social science programme

“…The correlation between prediction market beliefs and replication outcomes was 0.42 (RPP), 0.30 (EERP), 0.84 (SSRP) and 0.76 (ML2) in these four studies. If we carry out a power calculation for the average correlation of 0.58 of these four studies (as reported in [ 13 ]) on replication outcomes for the first and third primary hypotheses below, we have 51% power to detect a correlation of 0.58 at the 0.005 level for ‘statistical significance’ and 83% power at the 0.05 level for ‘suggestive evidence’. In another project (the crowdsourcing a hypothesis test (CAHT) project [ 20 ]) we used an incentivized survey to predict effect sizes for new studies and we observed a correlation of 0.71 between predicted and observed effect sizes; this previous result is relevant as a meter of comparison for the second and fourth primary hypotheses below.…”

“…As reported in Gordon et al . [ 13 ], pooling the results from four prediction market studies on binary outcomes (whether the study replicated or not; table 1 ) gives a 73% (76/104) correct prediction rate if we interpret prices above 50 as market prediction for successful replication. The corresponding number for a survey measure similar to the one we use here is 66% (68/103).…”

Establishing ground truth in the clinical neurosciences: if replication is the answer, then what are the questions?

“…have made use of prediction markets [38][39][40] which have been shown to be one of the best methods for predicting replicability, although they are quite costly and time consuming to run. Both approaches show enormous potential to advance reproducibility efforts.…”

A Synthetic Prediction Market for Estimating Confidence in Published Work

“…Given the challenges and significant resources required to run high-powered replication studies, researchers have sought other approaches to assess confidence in published claims and have looked to creative assembly of expert judgement as one opportunity. Initial evidence has supported the promise of prediction markets in this context (Dreber et al 2015;Camerer et al 2016Camerer et al , 2018Forsell et al 2019;Gordon et al 2020Gordon et al , 2021. However, practical deployment of prediction markets to evaluate scientific findings is also limited.…”