Confidence Calibration in a Multi-year Geopolitical Forecasting Competition

Abstract

This research examines the development of confidence and accuracy over time in the context of forecasting. Although overconfidence has been studied in many contexts, little research examines its progression over long periods of time or in consequential policy domains. This study employs a unique data set from a geopolitical forecasting tournament spanning three years in which thousands of forecasters predicted the outcomes of hundreds of events. We sought to apply insights from research to structure the questions, interactions, and elicitations to improve forecasts. Indeed, forecasters’ confidence roughly matched their accuracy. As information came in, accuracy increased. Confidence increased at approximately the same rate as accuracy, and good calibration persisted. Nevertheless, there was evidence of a small amount of overconfidence (3%), especially on the most confident forecasts. Training helped reduce overconfidence and team collaboration improved forecast accuracy. Together, teams and training reduced overconfidence to 1%. Our results provide reason for tempered optimism regarding confidence calibration and its development over time in consequential field contexts.

I nvestigate D iscuss E stimate A ggregate for structured expert judgement

“…We speculate that the most likely cause is a decrease in engagement. We note that this finding is in contrast to that of Mellers et al (2015a), who observed consistent improvements in participant calibration from year to year over the tournament, though they also found that more years of forecasting experience did not correspond to better performance relative to those of less experienced performers (Moore et al, 2015b).…”

“…Overconfidence (underconfidence) occurs when the observed relative frequency is lower (higher) than the subjective probability. While any increase in a participant's confidence would ideally be matched with an increase in performance, research across many different measures of confidence and performance has found that individuals' subjective confidence is often poorly correlated with their performance (e.g., Aspinall & Cooke, 2013;Griffin & Tversky, 1992;Lichtenstein et al, 1982;Moore et al, 2015b).…”

“…Crucial elements of the IDEA protocol were first formulated, combined and used by Burgman et al (2011);Speirs-Bridge et al (2010) and McBride et al (2012). However, the literature offers many competing recommendations as to the best ways of eliciting judgements and countering biases in predictive settings (e.g., Larrick, 2004;Moore et al, 2015b;Soll, Milkman, & Payne, 2015). More comparisons with alternative formulations of knowledge-resampling and group interactions would be beneficial.…”

The Value of Performance Weights and Discussion in Aggregated Expert Judgments

“…Ideally, any increase in an expert's confidence would be matched with an increase in his or her accuracy or calibration. However, research across many different measures of confidence, accuracy, and calibration has found that these measures are often poorly correlated with one another (e.g., Aspinall & Cooke, ; Griffin & Tversky, ; Lichtenstein et al., ; Moore et al., ).…”