Use (and abuse) of expert elicitation in support of decision making for public policy

Abstract

The elicitation of scientific and technical judgments from experts, in the form of subjective probability distributions, can be a valuable addition to other forms of evidence in support of public policy decision making. This paper explores when it is sensible to perform such elicitation and how that can best be done. A number of key issues are discussed, including topics on which there are, and are not, experts who have knowledge that provides a basis for making informed predictive judgments; the inadequacy of only using qualitative uncertainty language; the role of cognitive heuristics and of overconfidence; the choice of experts; the development, refinement, and iterative testing of elicitation protocols that are designed to help experts to consider systematically all relevant knowledge when they make their judgments; the treatment of uncertainty about model functional form; diversity of expert opinion; and when it does or does not make sense to combine judgments from different experts. Although it may be tempting to view expert elicitation as a low-cost, low-effort alternative to conducting serious research and analysis, it is neither. Rather, expert elicitation should build on and use the best available research and analysis and be undertaken only when, given those, the state of knowledge will remain insufficient to support timely informed assessment and decision making.

Cost-benefit analysis of foot and mouth disease control in Ethiopia

“…Some experts expected RVS to have even a negative effect on the incidence reduction. Because of the widely different judgments, it was found inappropriate to linearly pool their judgments to define one overall probability distribution by averaging the parameter entries among experts (Keith, 1996;Morgan, 2014). The individual experts' pert probability distributions were rather combined to a single composite distribution using a discrete uniform probability distribution with equal weights as described by Vose (2008).…”

An expert elicitation process to project the frequency and magnitude of Florida manatee mortality events caused by red tide (Karenia brevis)

“…Because such complex models are not currently available, we used an expert judgment process (Morgan, 2014;Lyon and others, 2015;Sutherland and Burgman, 2015). Expert judgment techniques can be used to elicit probability distributions from a group of experts when information about the parameters of interest is sparse or nonexistent (Morgan, 2014;Lyon and others, 2015). Formal expert judgment methods use a structured approach to elicit parameter estimates from experts while accounting for uncertainty, reducing cognitive biases, and integrating the best available scientific information.…”

“…Formal expert judgment methods use a structured approach to elicit parameter estimates from experts while accounting for uncertainty, reducing cognitive biases, and integrating the best available scientific information. Expert judgment is used extensively for decision making in many settings, such as health assessment, evaluation of the impacts of global climate change, engineering, and natural resource management (Morgan, 2014;Lyon and others, 2015;Sutherland and Burgman, 2015). The baseline scenario of the CBM used the expert-elicited estimates of future frequency and magnitude of RTMM events because most experts believed that these estimates represented better reflections of the future than did the estimates based on historical patterns.…”

“…We emphasized that the goal of the second round was not to reach a consensus but to reduce errors associated with potential miscommunication (that is, linguistic uncertainty) and to further clarify the mechanisms of manatee mortality associated with red tides (Sutherland and Burgman, 2015). This approach differs from the more traditional use of the Delphi method, in which experts are encouraged to reach a consensus (Morgan, 2014;Johnson and others, 2017). One limitation of seeking consensus is that it may result from group pressure and, therefore, may mischaracterize the actual state of knowledge (Woudenberg, 1991;Morgan, 2014;Burgman, 2015, Johnson andothers, 2017).…”

“…This approach differs from the more traditional use of the Delphi method, in which experts are encouraged to reach a consensus (Morgan, 2014;Johnson and others, 2017). One limitation of seeking consensus is that it may result from group pressure and, therefore, may mischaracterize the actual state of knowledge (Woudenberg, 1991;Morgan, 2014;Burgman, 2015, Johnson andothers, 2017).…”