Hidden Experts in the Crowd: Using Meta-Predictions to Leverage Expertise in Single-Question Prediction Problems

Abstract

Modern forecasting algorithms use the wisdom of crowds to produce forecasts better than those of the best identifiable expert. However, these algorithms may be inaccurate when crowds are systematically biased or when expertise varies substantially across forecasters. Recent work has shown that meta-predictions—a forecast of the average forecasts of others—can be used to correct for biases even when no external information, such as forecasters’ past performance, is available. We explore whether meta-predictions can also be used to improve forecasts by identifying and leveraging the expertise of forecasters. We develop a confidence-based version of the Surprisingly Popular algorithm proposed by Prelec, Seung, and McCoy. As with the original algorithm, our new algorithm is robust to bias. However, unlike the original algorithm, our version is predicted to always weight forecasters with more informative private signals more than forecasters with less informative ones. In a series of experiments, we find that the modified algorithm does a better job in weighting informed forecasters than the original algorithm and show that individuals who are correct more often on similar decision problems contribute more to the final decision than other forecasters. Empirically, the modified algorithm outperforms the original algorithm for a set of 500 decision problems. This paper was accepted by Yan Chen, decision analysis.

Extracting the collective wisdom in probabilistic judgments

“…When shared information differs from the true probability, experts are likely to err in the same direction, resulting in a miscalibrated average prediction. The SO algorithm relies on an augmented elicitation proposed in recent work (Prelec, 2004;Prelec et al 2017;Palley and Soll 2019;Palley and Satopää 2022;Wilkening et al 2022): Experts report a prediction of the probability as well as an estimate of the average of others' predictions, which is referred to as a meta-prediction. I show that when the average prediction is a consistent estimator, the percentage of predictions and metapredictions that overshoot the average prediction should be the same.…”

“…Their experiment implements shared and private signals as sample flips from the biased coin. The second source is Wilkening et al (2022), who conducted two experimental studies. The first experiment replicates the earlier study by Prelec et al (2017) which asked subjects true/false questions about the capital cities of U.S. states.…”

“…However, unlike Prelec et al (2017) they also ask subjects to report probabilistic predictions and metapredictions, which allows an implementation of the SO algorithm. In the second experiment, Wilkening et al (2022) generate 500 basic science statements and ask subjects to report probabilistic predictions and meta-predictions on the likelihood that a given statement is true. Results suggest that the SO algorithm outperforms simple benchmarks such as unweighted averaging and median prediction.…”

“…Results suggest that the SO algorithm outperforms simple benchmarks such as unweighted averaging and median prediction. I also compare the SO algorithm to alternative solutions for aggregating probabilistic judgments, which elicit similar information from individuals (Palley and Soll 2019;Martinie et al 2020;Palley and Satopää, 2022;Wilkening et al 2022). The SO algorithm compares favorably to alternative aggregation mechanisms in prediction tasks where individual predictions are highly dispersed.…”

“…The Surprisingly Popular (SP) algorithm picks an answer to a multiple choice question based on predicted and realized endorsement rates of alternative choices (Prelec et al 2017). The Surprisingly Confident (SC) algorithm determines weights that leverage more informed judgments (Wilkening et al 2022). The SP and SC algorithms aim to find the correct answer to a binary or multiple-choice question while the SO algorithm produces a probabilistic estimate on a binary event.…”