Using prediction markets to estimate the reproducibility of scientific research

Abstract

Concerns about a lack of reproducibility of statistically significant results have recently been raised in many fields, and it has been argued that this lack comes at substantial economic costs. We here report the results from prediction markets set up to quantify the reproducibility of 44 studies published in prominent psychology journals and replicated in the Reproducibility Project: Psychology. The prediction markets predict the outcomes of the replications well and outperform a survey of market participants' individual forecasts. This shows that prediction markets are a promising tool for assessing the reproducibility of published scientific results. The prediction markets also allow us to estimate probabilities for the hypotheses being true at different testing stages, which provides valuable information regarding the temporal dynamics of scientific discovery. We find that the hypotheses being tested in psychology typically have low prior probabilities of being true (median, 9%) and that a "statistically significant" finding needs to be confirmed in a well-powered replication to have a high probability of being true. We argue that prediction markets could be used to obtain speedy information about reproducibility at low cost and could potentially even be used to determine which studies to replicate to optimally allocate limited resources into replications.reproducibility | replications | prediction markets T he process of scientific discovery centers on empirical testing of research hypotheses. A standard tool to interpret results in statistical hypothesis testing is the P value. A result associated with a P value below a predefined significance level (typically 0.05) is considered "statistically significant" and interpreted as evidence in favor of a hypothesis. However, concerns about the reproducibility of statistically significant results have recently been raised in many fields including medicine (1-3), neuroscience (4), genetics (5, 6), psychology (7-11), and economics (12, 13). For example, an industrial laboratory could only reproduce 6 out of 53 key findings from "landmark" studies in preclinical oncology (2) and it has been argued that the costs associated with irreproducible preclinical research alone are about US$28 billion a year in the United States (3). The mismatch between the interpretation of statistically significant findings and a lack of reproducibility threatens to undermine the validity of statistical hypothesis testing as it is currently practiced in many research fields (14).The problem with inference based on P values is that a P value provides only partial information about the probability of a tested hypothesis being true (14,15). This probability also depends on the statistical power to detect a true positive effect and the prior probability that the hypothesis is true (14). Lower statistical power increases the probability that a statistically significant effect is a false positive (4, 14). Statistically significant results from small studies are therefore more likely to be fals...

Evaluating replicability of laboratory experiments in economics

“…The absolute prediction error does not differ significantly between the prediction market (Mean=0.414) and the pre-market survey (Mean=0.409) (Wilcoxon signed-rank test, n=18, z=0.33, P=0.744). Contrary to a recent prediction market study on a subset of the studies (n=44) included in the RPP project (26), the prediction market thus does not predict replication outcomes better than the survey. However, the sample size of replications is small with only 18 observations.…”

“…Almenberg et al (58) conducted a lab-based test. More recently a prediction market study on replications in psychology has yielded promising results (26), in the sense that predictions revealed by market prices are correlated with actual replication outcomes (and are more strongly correlated than surveyed beliefs). Prediction markets have also been successfully used in several other fields such as sports, entertainment, and politics (25,(59)(60)(61)(62).…”

“…The prediction market methodology used in this study is similar to the one used in Dreber et al (26). Dreber et al (26) presented prediction market results for 44 studies in RPP.…”

“…Dreber et al (26) presented prediction market results for 44 studies in RPP. The trading platforms and the participant pool differed between the two prediction market studies (a sample of psychologists participated in the Dreber et al (26) study and a sample of economists, see above, participated in this study).…”

Contextual sensitivity in scientific reproducibility

“…Numerous other suggestions for improving reproducibility have been proposed (e.g., refs. 62,76,86,87). For example, the replication recipe (76) offers a "five-ingredient" approach to standardizing replication attempts that emphasizes precision, power, transparency, and collaboration.…”