A Comparison of Two Probability Encoding Methods: Fixed Probability vs. Fixed Variable Values

Abstract

W e present the results of an experiment comparing two popular methods for encoding probability distributions of continuous variables in decision analysis: eliciting values of a variable, X, through comparisons with a fixed probability wheel and eliciting the percentiles of the cumulative distribution, F X , through comparisons with fixed values of the variable. We show slight but consistent superiority for the fixed variable method along several dimensions such as monotonicity, accuracy, and precision of the estimated fractiles. The fixed variable elicitation method was also slightly faster and preferred by most participants. We discuss the reasons for its superiority and conclude with several recommendations for the practice of probability assessment.

Testing Best Practices to Reduce the Overconfidence Bias in Multi-criteria Decision Analysis

“…Abbas et al [12] and Seaver et al [13] showed that participants produce less confidence in their accuracy when they were asked to provide probabilities for fixed values of the random variables than when they were asked to specify values for fixed cumulative probabilities. Teigen and Jørgensen [ 44 ] demonstrated that people have less confidence when they estimate values for a pre-determined confidence level.…”

“…For example, weather forecasters, who have much experience with probability judgments and receive feedback almost daily, are remarkably well calibrated (for a summary, see von Winterfeldt and Edwards [11]). Regarding the response mode, Abbas et al, [12] and Seaver et al [13] compared the usual fixed probability method (providing values for fixed cumulative probabilities) with a method in which the participants provided cumulative probabilities for fixed values and found better calibration for the latter.…”

From the Editors: Probability Approximations, Anti-Terrorism Strategy, and Bull's-Eye Display for Performance Feedback

“…Prior Decision Analysis papers on probability assessment by other authors include Abbas et al (2008), Baillon (2008) on using exchangeable events, Bordley (2011) on updating probabilities based on outcomes of partially similar events, and Bordley (2009) on experts who partition events differently. Kilgour and Gerchak (2004), Johnstone (2007), and Schervish et al (2009) have published papers in Decision Analysis on probability scoring rules, used to judge the accuracy of assessed probabilities.…”

From the Editors—Deterrence, Multiattribute Utility, and Probability and Bayes' Updating

“…A related prior contribution in Decision Analysis is Bordley (2009) on combining the opinions of experts who partition events differently. Related papers in Decision Analysis include Abbas et al (2008) on two probability assessment methods; Baillon (2008) on a method for eliciting probabilities using exchangeable events; and Bickel (2010), Johnstone (2007), Kilgour and Gerchak (2004), and Schervish et al (2009) on probability scoring rules. Bordley provided leadership as a guest editor for the Michael H. Rothkopf special issue (see Bordley et al 2010) and is a member of the journal's editorial board.…”