Talent Spotting in Crowd Prediction

Abstract

Who is good at prediction? Addressing this question is key to recruiting and cultivating accurate crowds and effectively aggregating their judgments. Recent research on superforecasting has demonstrated the importance of individual, persistent skill in crowd prediction. This chapter takes stock of skill identification measures in probability estimation tasks, and complements the review with original analyses, comparing such measures directly within the same dataset. We classify all measures in five broad categories: 1) accuracy-related measures, such as proper scores, model-based estimates of accuracy and excess volatility scores; 2) intersubjective measures, including proxy, surrogate and similarity scores; 3) forecasting behaviors, including activity, belief updating, extremity, coherence, and linguistic properties of rationales; 4) dispositional measures of fluid intelligence, cognitive reflection, numeracy, personality and thinking styles; and 5) measures of expertise, including demonstrated knowledge, confidence calibration, biographical, and self-rated expertise. Among non-accuracy-related measures, we report a median correlation coefficient with outcomes of r = 0.20. In the absence of accuracy data, we find that intersubjective and behavioral measures are most strongly correlated with forecasting accuracy. These results hold in a LASSO machine-learning model with automated variable selection. Two focal applications provide context for these assessments: long-term, existential risk prediction and corporate forecasting tournaments.

The wisdom of many in few: Finding individuals who are as wise as the crowd.

“…Because it is the first controlled longitudinal forecasting experiment, future work could use this dataset to firmly establish the degree of bias introduced when these confounds are artificially induced and various existing methods are used to attempt to eliminate them. For example, subsets of forecasters can be deleted from the dataset based on patterns of question difficulty, such that some forecasters are retained for difficult questions, and others for harder questions, to study how well within-item score standardization (Atanasov et al, 2017, 2020; Atanasov & Himmelstein, 2022; Himmelstein et al, 2021) adjusts for item difficulty in practice. Similarly, forecasters can be deleted from the dataset as a function of the timing of their forecasts, to assess temporal confounds in judgmental forecasting theory (Himmelstein, Budescu & Han, 2022).…”

“…Past research exploring the utility of this approach has proved fruitful but methodologically challenging. Most research on the psychology of forecasting utilizes one of two sources of data: (a) judgments collected during forecasting tournaments, in which large samples of forecasters self-select from among hundreds of possible forecasting questions, and make their predictions at different points in time, at their leisure (Atanasov & Himmelstein, 2022; Himmelstein et al, 2021; Mellers, Stone, Atanasov, et al, 2015; Morstatter et al, 2019; Tetlock & Gardner, 2016); (b) professional forecasters, who make predictions for a living about finance, sports, weather, health, politics, and more (Garcia, 2003; Han & Budescu, 2019, 2022; Himmelstein, Budescu & Han, 2022; Mandel & Barnes, 2014; Spann & Skiera, 2009). Both data sources inherently include several confounds that make controlled scientific inference difficult.…”

“…Several methods have been proposed for addressing endogenous question selection and forecast timing, which demonstrably improve forecaster evaluation methods when these confounds are a potential nuisance (Atanasov & Himmelstein, 2022; Himmelstein et al, 2021; Himmelstein, Budescu & Han, 2022; Mellers et al, 2014; Mellers, Stone, Atanasov, et al, 2015). However, no study has ever been conducted that fully controls for these factors experimentally.…”

“…Nonrandom assignment also occurs in research involving professional forecasters, who are assigned to their tasks based on their skillsets and expertise. Accounting for self-selection, forecast timing, and differences in question difficulty are all recognized as important confounds that need to be accounted for in forecaster evaluation research (Himmelstein, Budescu & Han, 2022;Himmelstein et al, 2021) Several methods have been proposed for addressing endogenous question selection and forecast timing, which demonstrably improve forecaster evaluation methods when these confounds are a potential nuisance (Atanasov & Himmelstein, 2022;Himmelstein et al, 2021;Himmelstein, Budescu & Han, 2022;Mellers et al, 2014;Mellers, Stone, Atanasov, et al, 2015). However, no study has ever been conducted that fully controls for these factors experimentally.…”

“…Another forecaster may ignore that question entirely, and only make predictions about a highly contested upcoming election, which is much more difficult to predict. When this happens, differences in question difficulty must be accounted for to compare the quality of the two forecasters' predictions (Atanasov & Himmelstein, 2022;Bo et al, 2017;Himmelstein et al, 2021). A third forecaster might forecast the results of the same election as the second, but wait several weeks, when one of the candidates surges in the polls, giving them an informational advantage, which likewise must be accounted for (Himmelstein, Budescu & Han, 2022;Himmelstein et al, 2021).…”