Probability Elicitation Under Severe Time Pressure: A Rank‐Based Method

Abstract

Probability elicitation protocols are used to assess and incorporate subjective probabilities in risk and decision analysis. While most of these protocols use methods that have focused on the precision of the elicited probabilities, the speed of the elicitation process has often been neglected. However, speed is also important, particularly when experts need to examine a large number of events on a recurrent basis. Furthermore, most existing elicitation methods are numerical in nature, but there are various reasons why an expert would refuse to give such precise ratio-scale estimates, even if highly numerate. This may occur, for instance, when there is lack of sufficient hard evidence, when assessing very uncertain events (such as emergent threats), or when dealing with politicized topics (such as terrorism or disease outbreaks). In this article, we adopt an ordinal ranking approach from multicriteria decision analysis to provide a fast and nonnumerical probability elicitation process. Probabilities are subsequently approximated from the ranking by an algorithm based on the principle of maximum entropy, a rule compatible with the ordinal information provided by the expert. The method can elicit probabilities for a wide range of different event types, including new ways of eliciting probabilities for stochastically independent events and low-probability events. We use a Monte Carlo simulation to test the accuracy of the approximated probabilities and try the method in practice, applying it to a real-world risk analysis recently conducted for DEFRA (the U.K. Department for the Environment, Farming and Rural Affairs): the prioritization of animal health threats.

A Risk Analysis Framework for Prioritizing and Managing Biosecurity Threats

“…Most of these models assume deterministic impacts employing different multicriteria methods such as: multiattribute value theory (e.g., the prioritization of exotic diseases for the pig industry in Australia by Brookes et al., 2014 and the prioritization of animal health threats to the United Kingdom by Del Rio Vilas, Voller, et al., 2013); the analytic hierarchy process (e.g., the stakeholder prioritization of zoonoses in Japan by Kadohira, Hill, Yoshizaki, Ota, & Yoshikawa, 2015 and the prioritization of zoonotic diseases in Kenya by Munyua et al., 2016); or score‐and‐weight models without decision analysis roots (e.g., the prioritization of communicable diseases in Germany by Balabanova et al., 2011 and the prioritization of diseases and zoonoses in Europe by Humblet et al., 2012). In general, these models neglect the probabilistic component of the problem, adopting a precautionary principle and thus making the implicit assumption that threats have all the same likelihood of occurrence (Jaspersen & Montibeller, 2015); they also assume deterministic impacts, instead of defining distributions that represent their uncertainty as is standard in risk analysis (Morgan & Henrion, 1992).…”

“…Additionally, the spatial separation and time lags between benefits obtained from risk mitigation actions and costs associated with such actions increase the complexity of these assessments. Fourth, there is typically a limited amount of time to gather hard evidence about possible impacts and likelihood of occurrence of outbreaks for each of the threats, as well as limited resources for building up extensive predictive models that can simulate their transmission patterns and impacts (Del Rio Vilas, Voller, et al., 2013; Jaspersen & Montibeller, 2015). Furthermore, learning from past projects is difficult, due to the disconnect between decisions made and their consequences in the absences of counterfactuals (Roese, 1999), that is, knowing what would have happened if a different risk mitigation action were implemented, or if no risk mitigation were implemented at all (see also Woo, 2016 for a detailed discussion on counterfactuals for risk analysis).…”

Measuring Conflicts Using Cardinal Ranking: An Application to Decision Analytic Conflict Evaluations

“…The CAR method has been demonstrated using both linear inequalities to represent cardinal ranking statements [34] and closed formulas for obtaining surrogate weights [10,30]. More recently, rank-based methods have been suggested for probability elicitation as well, with a particular aim for use in time scarce environments [35].…”

Improving occupational health and safety risk evaluation through decision analysis

“…There is thus space for exploring distinct methods to elicit subjective probabilities. In line with other studies that have explored the elicitation of probabilities using ordinal nonnumerical probability elicitation process (Jaspersen and Montibeller, ), qualitative assessments (Monti and Carenini, ) and pairwise comparisons (Por and Budescu, ) and within the methodological coherence that characterizes the IRIS approach (that redesigns and explores RMs with MACBETH), a system for the OHSU to define subjective probabilities for distinct OHS risks based on MACBETH was devised (a detailed description of this work is reported in Bernardes, ).…”