Reducing Overconfidence in the Interval Judgments of Experts

Abstract

Elicitation of expert opinion is important for risk analysis when only limited data are available. Expert opinion is often elicited in the form of subjective confidence intervals; however, these are prone to substantial overconfidence. We investigated the influence of elicitation question format, in particular the number of steps in the elicitation procedure. In a 3-point elicitation procedure, an expert is asked for a lower limit, upper limit, and best guess, the two limits creating an interval of some assigned confidence level (e.g., 80%). In our 4-step interval elicitation procedure, experts were also asked for a realistic lower limit, upper limit, and best guess, but no confidence level was assigned; the fourth step was to rate their anticipated confidence in the interval produced. In our three studies, experts made interval predictions of rates of infectious diseases (Study 1, n = 21 and Study 2, n = 24: epidemiologists and public health experts), or marine invertebrate populations (Study 3, n = 34: ecologists and biologists). We combined the results from our studies using meta-analysis, which found average overconfidence of 11.9%, 95% CI [3.5, 20.3] (a hit rate of 68.1% for 80% intervals)-a substantial decrease in overconfidence compared with previous studies. Studies 2 and 3 suggest that the 4-step procedure is more likely to reduce overconfidence than the 3-point procedure (Cohen's d = 0.61, [0.04, 1.18]).

Policy advice: Use experts wisely

“…For example, Andrew Speirs-Bridge at La Trobe University has shown 9 that questions that elicit four responses -upper and lower bounds, a best guess and a degree of confidence in the interval -generate estimates that are relatively well calibrated. Consider a range of scenarios and alternative theories.…”

Overconfidence of Students and Managers - Comparative Analysis

“…Researchers have offered overconfidence as an explanation for medical misdiagnosis, mistaken court judgement based on overconfident witnesses' statements, road accidents, politicians' decisions (Johnson, 2004). Overconfidence has been confirmed also among epidemiologists, public health experts, ecologists, and biologists (Speirs-Bridge et al, 2010).…”

Stochastic Population Forecasting Based on Combinations of Expert Evaluations Within the Bayesian Paradigm

“…Additional research is undoubtedly needed on how to better gain knowledge from experts in human population forecasting, including, in particular, efforts to limit their overconfidence. This issue has been tackled in other areas of forecasting, in fields as diverse as politics, economics and finance epidemiology, and population ecology (e.g., Speirs-Bridge et al 2010;Tetlock 2005;Tyszka and Zielonka 2002).…”