Improving Accuracy by Coherence Weighting of Direct and Ratio Probability Judgments

Abstract

Human forecasts and other probabilistic judgments can be improved by elicitation and aggregation methods. Recent work on elicitation shows that deriving probability estimates from relative judgments (the ratio method) is advantageous, whereas other recent work on aggregation shows that it is beneficial to transform probabilities into coherent sets (coherentization) and to weight judges' assessments by their degree of coherence. We report an experiment that links these areas by examining the effect of coherentization and multiple forms of coherence weighting using direct and ratio elicitation methods on accuracy of probability judgments (both forecasts and events with known distributions). We found that coherentization invariably yields improvements to accuracy. Moreover, judges' levels of probabilistic coherence are related to their judgment accuracy. Therefore, coherence weighting can improve judgment accuracy, but the strength of the effect varies among elicitation and weighting methods. As well, the benefit of coherence weighting is stronger on “calibration” items that served as a basis for establishing the weights than for unrelated “test” items. Finally, echoing earlier research, we found overconfidence in judgment, and the degree of overconfidence was comparable between the two elicitation methods.

Improving Probability Judgment in Intelligence Analysis: From Structured Analysis to Statistical Aggregation

“…We explored competing analytic methods, including ACH and FBT, and competing post-analytic methods, including coherentization using different constraints as well as equal-weighted aggregation versus a variety of coherenceweighted aggregation. Consistent with Fan et al (2019), our findings indicate that researchers and practitioners alike who seek to improve judgment accuracy should consider the performance of ensembles of methods. Consider ACH: on its own it was the least effective method, actually reducing accuracy and coherence compared to a within-subject no-method control.…”

“…In this research, using multiple constraints-namely, unitarity and additivity-paid off (see also Karvetski et al, 2013b). However, other studies have found better results when only the unitarity constraint is applied (Fan et al, 2019;. Although we cannot explain this variability at this time, it is clear that the largest effect is between any form of coherentization and none.…”

“…In particular, we aimed to investigate accuracy sensitivity for different methods of measuring coherence and incorporating coherence-weighting in aggregation (cf. Fan et al, 2019). Figure 4 shows MAE by group size when coherence weighting of U-form post-method direct judgments is applied within the ACH and FBT groups for a) different scale values of β (different panels) and with b) weights derived from different estimates and/or coherence schemes (different line colors).…”

“…While some weighting schemes rely on having seed data with known resolutions to assess the quality of individuals (Cooke, 1990;Cooke & Goossens, 2008;Van der Fels-Klerx, Cooke, Nauta, Goossens, & Havelaar, 2005), other methods assign greater weight to more coherent individuals and therefore do not require seed data (Karvetski et al, 2013b;Wang, Kulkarni, Poor, & Osherson, 2011). Nevertheless, when judged against the equal weighting of individuals, the merit of performance-based weighting schemes is not without question (Bolger & Rowe, 2014;Cooke, 2015;Fan, Budescu, Mandel, & Himmelstein, 2019).…”

From the Editors: Decision Analysis Focus and Trends

“…Contributions on elicitation and aggregation can range from behavioral experiments (Fan et al 2019, Schneider et al 2019 to mathematical methods or heuristics (Huang and Bickel 2019). Such papers could have been characterized as methodological or behavioral, but we highlight them in a separate category because elicitation is of central importance to the practice and continued improvement of expected-utility decision theory.…”