The consistency, coherence and calibration of holistic, decomposed and recomposed judgemental probability forecasts

Abstract

In this paper we make an empirical investigation of the relationship between the consistency, coherence and validity of probability judgements in a real-world forecasting context. Our results indicate that these measures of the adequacy of an individual's probability assessments are not closely related as we anticipated. Twenty-nine of our thirty-six subjects were better calibrated in point probabilities than in odds and our subjects were, in general more coherent using point probabilities than odds forecasts. Contrary to our expectations we found very little difference in forecasting response and performance between simple and compound holistic forecasts. This result is evidence against the 'divide-and-conquer' rationale underlying most applications of normative decision theory. In addition, our recompositions of marginal and conditional assessments into compound forecasts were no better calibrated or resolved than their holistic counterparts.These findings convey two implications for forecasting. First, untrained judgemental forecasters should use point probabilities in preference to odds. Second, judgemental forecasts of complex compound probabilities may be as well assessed holistically as they are using methods of decomposition and recomposition. In addition, our study provides a paradigm for further studies of the relationship between consistency, coherence and validity in judgemental probability forecasting. KEY WORDS Probability Judgements Judgemental ForecastingCalibration Cross-impact AnalysisJudgemental probabilities are prime inputs to many management decision-aiding technologies such as decision analysis, cross-impact analysis and fault-tree analysis. These subjective probabilities are, essentially, probability forecasts of the likelihood of occurrence of future events. Psychological studies of subjective probability estimation have tended to involve

Expertise, credibility of system forecasts and integration methods in judgmental demand forecasting

“…However, the lack of information availability might offset this advantage. Although this method has been suggested for forecasting tasks (Jones, Wheeler, Appan, & Saleem, 2006;Wright, Saunders, & Ayton, 1988), its applicability to the specific case of demand forecasting has not been tested.…”

Expert Judgement of Probability and Risk

“…Unfortunately, there is little empirical evidence available to answer this question. Wright et al (1988) found that calibration of unions, intersection and disjunctions of events which were mechanically recomposed from marginal and conditional assessments were no better calibrated than holistically judged forecast probabilities for the same events. Further, no significant correlation was found between the coherence and the subsequent calibration of the assessed probabilities.…”

“…Depending on which metric is used, the judge's task of turning subjective feelings of uncertainty into measurable/usable numeric estimates can be either helped or hindered. For example, Wright et al (1988) found that for a short-term forecasting task of impersonal events (e.g. "will the pound fall below one dollar in the next 2 months?…”

Peer Input and Revised Judgment: Exploring the Effects of (Un) Biased Confidence1

“…That is, assigned probabilities generally exceed the proportion of judgments that turn out to be correct (for reviews, see Lichtenstein, Fischhoff, & Phillips, 1982;Yates, 1990). This bias has been observed on a wide variety of tasks involving general knowledge (e.g., Koriat, Lichtenstein, & Fischhoff, 1980;Paese & Feuer, 1991;Yates et at., 1989), interpersonal judgment (e.g., Dunning, Griffin, Milojkovic, & Ross, 1990;Paese & Kinnaly, 1991), and forecasts of personal events (e.g., Hoch, 1985;Vallone, Griffin, Lin, & Ross, 1990;Wright & Ayton, 1989), and impersonal events (e.g., Fischhoff & MacGregor, 1982;Wright, Saunders, & Ayton, 1988). Although there are some exceptions to this bias (see Keren, 1987;Murphy & Winkler, 1984), there is little reason to believe that peers are any less prone to overconfidence when sharing their judgments on difficult matters.…”