The wisdom of many in few: Finding individuals who are as wise as the crowd.

Abstract

Is forecasting ability a stable trait? While domain knowledge and reasoning abilities are necessary for making accurate forecasts, research shows that knowing how accurate forecasters have been in the past is the best predictor of future accuracy. However, unlike the measurement of other traits, evaluating forecasting skill requires substantial time investment. Forecasters must make predictions about events that may not resolve for many days, weeks, months, or even years into the future before their accuracy can be estimated. Our work builds upon methods such as cultural consensus theory and proxy scoring rules to show talented forecasters can be discriminated in real time, without requiring any event resolutions. We define a peer similarity-based intersubjective evaluation method and test its utility in a unique longitudinal forecasting experiment. Because forecasters predicted all events at the same points in time, many of the confounds common to forecasting tournaments or observational data were eliminated. This allowed us to demonstrate the effectiveness of our method in real time, as time progressed and more information about forecasters became available. Intersubjective accuracy scores, which can be obtained immediately after the forecasts are made, were both valid and reliable estimators of forecasting talent. We also found that asking forecasters to make meta-predictions about what they expect others to believe can serve as an incentive-compatible method of intersubjective evaluation. Our results indicate that selecting small groups of, or even single forecasters, based on intersubjective accuracy can yield subsequent forecasts that approximate the actual accuracy of much larger crowd aggregates.

Using selected peers to improve the accuracy of crowd sourced forecasts.

“…We stress again, that the imputation does not require information about the ground truth and the past accuracy of the selected peers. Thus, it can be implemented as soon as the forecasting ends and does not require waiting for the revelation of the true state of the world (see also Himmelstein et al, 2023).…”

Talent Spotting in Crowd Prediction

“…A related variation, the expected Brier Score (EBS), is the average Brier score across each possible outcome weighted by the probability the crowd assigns to those outcomes (Himmelstein, Budescu & Ho, 2022) Bayesian Truth Serum: This method was originally developed by Prelec (2004) and also applies to both resolvable and unresolvable questions. Respondents are asked about their own best guess about the true answer, as well as their estimate of the crowd's average answer.…”

“…Himmelstein, Budescu and Ho (2022) expanded on this work in a study in which 175 forecasters made predictions about 11 events related to politics, economics, and public health. Each forecaster forecasted each event five times at three-week intervals leading up to event resolution.…”

Boosting wisdom of the crowd for medical image annotation using training performance and task features

“…However, it is not clear how many people one must retain during the aggregation process. Tetlock and Gardner ( 2016 ) and Himmelstein et al ( 2023 ) argue for the existence of superforecasters, who if identified can consistently beat the crowd. In Goldstein et al ( 2014 ), the authors find a decreasing relationship with the number of experts, where the performance decreases as more individuals are included.…”