Probabilistic forecasting of replication studies

Pawel, Samuel; Held, Leonhard

doi:10.1371/journal.pone.0231416

Cited by 31 publications

(29 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The data and results presented in this paper can be used for future forecasting projects that are either planned or in progress [ 14 ], by informing experimental design and forecasting aggregation. The results can also be used to evaluate the predictive performance of prediction markets against other methods [ 33 , 34 , 40 ]. The pooled dataset presents opportunities for other researchers investigate replicability of scientific research, human forecasts and their intersection, as well as providing a benchmark for any further replication-based markets.…”

Section: Discussionmentioning

confidence: 99%

Predicting replicability—Analysis of survey and prediction market data from large-scale forecasting projects

et al. 2021

View full text Add to dashboard Cite

The reproducibility of published research has become an important topic in science policy. A number of large-scale replication projects have been conducted to gauge the overall reproducibility in specific academic fields. Here, we present an analysis of data from four studies which sought to forecast the outcomes of replication projects in the social and behavioural sciences, using human experts who participated in prediction markets and answered surveys. Because the number of findings replicated and predicted in each individual study was small, pooling the data offers an opportunity to evaluate hypotheses regarding the performance of prediction markets and surveys at a higher power. In total, peer beliefs were elicited for the replication outcomes of 103 published findings. We find there is information within the scientific community about the replicability of scientific findings, and that both surveys and prediction markets can be used to elicit and aggregate this information. Our results show prediction markets can determine the outcomes of direct replications with 73% accuracy (n = 103). Both the prediction market prices, and the average survey responses are correlated with outcomes (0.581 and 0.564 respectively, both p < .001). We also found a significant relationship between p-values of the original findings and replication outcomes. The dataset is made available through the R package “pooledmaRket” and can be used to further study community beliefs towards replications outcomes as elicited in the surveys and prediction markets.

show abstract

Section: Discussionmentioning

confidence: 99%

Predicting replicability—Analysis of survey and prediction market data from large-scale forecasting projects

et al. 2021

View full text Add to dashboard Cite

show abstract

“…If these findings are themselves replicable, then machine learning algorithms could provide a highscalable early assessment of replicability and credibility to inform evaluation, resource allocation, and identification of gaps and strengths in the empirical evidence for theoretical models and findings. A third study used a different type of forecasting approach using the original studies' information and the replication studies' sample size only (Pawel & Held, 2020). For the comparable samples, the forecasts from the tested statistical methods performed as well as or worse than the prediction markets.…”

Section: Predicting Replicabilitymentioning

confidence: 99%

Replicability, Robustness, and Reproducibility in Psychological Science

Nosek¹,

Hardwicke²,

Moshontz³

et al. 2021

Preprint

View full text Add to dashboard Cite

Replication, an important, uncommon, and misunderstood practice, is making a comeback in psychology. Achieving replicability is a necessary but not sufficient condition for making research progress. If findings are not replicable, then prediction and theory development are stifled. If findings are replicable, then interrogation of their meaning and validity can advance knowledge. Assessing replicability can be productive for generating and testing hypotheses by actively confronting current understanding to identify weaknesses and spur innovation. For psychology, the 2010s might be characterized as a decade of active confrontation. Systematic and multi-site replication projects assessed current understanding and observed surprising failures to replicate many published findings. Replication efforts also highlighted sociocultural challenges, such as disincentives to conduct replications, framing of replication as personal attack rather than healthy scientific practice, and headwinds for replication contributing to self-correction. Nevertheless, innovation in doing and understanding replication, and its cousins, reproducibility and robustness, have positioned psychology to improve research practices and accelerate progress.

show abstract

“…Note that the amount of shrinkage of the prediction depends on the signal‐to‐noise ratio, which can be summarised by the p ‐value of the original study. The adjustment barely shrinks large effect estimates from very convincing original studies but applies more shrinkage to less convincing ones 6 . For our simulated data, the adjustment reduces the mean squared prediction error from 1.87 to 1.56 (17%).…”

Section: Publication Biasmentioning

confidence: 80%

Replication Power and Regression to The Mean

Held

Pawel²,

Schwab³

2020

Significance

Self Cite

View full text Add to dashboard Cite

If a scientific study reports a discovery with a p-value at or around 0.05, how credible is it? And what are the chances that a replication of this study will produce a similarly "significant" finding? Leonhard Held, Samuel Pawel and Simon Schwab's answers may surprise you One may wonder how such a large discrepancy between the original and the replication study could have occurred. One helpful approach to answer this question is the concept of replication power.

show abstract

Probabilistic forecasting of replication studies

Cited by 31 publications

References 45 publications

Predicting replicability—Analysis of survey and prediction market data from large-scale forecasting projects

Predicting replicability—Analysis of survey and prediction market data from large-scale forecasting projects

Replicability, Robustness, and Reproducibility in Psychological Science

Replication Power and Regression to The Mean

Contact Info

Product

Resources

About