A practical guide to structured expert elicitation using the IDEA protocol

Hemming, Victoria; Burgman, Mark A.; Hanea, Anca M.; McBride, Marissa; Wintle, Bonnie

doi:10.1111/2041-210x.12857

Cited by 301 publications

(404 citation statements)

References 62 publications

Supporting

Mentioning

397

Contrasting

Unclassified

Order By: Relevance

“…Elicitation processes should be iterative and allow for wide ranging discussion (e.g., Hemming, Burgman, Hanea, McBride, & Wintle, 2018). The conundrum we faced, created by a combination of objective and subjective uncertainties and risks, might be common for such programs.…”

Section: Discussionmentioning

confidence: 99%

Risk aversion and uncertainty create a conundrum for planning recovery of a critically endangered species

Canessa

Taylor

Clarke

et al. 2019

Conservat Sci and Prac

View full text Add to dashboard Cite

Making transparent and rational decisions to manage threatened species in situations of high uncertainty is difficult. Managers must balance the optimism of successful intervention with the risk that intervention could make matters worse. We assessed nest protection options for regent honeyeaters (Anthochaera phrygia) in Australia. Formal expert elicitation highlighted two methods of nest protection expected to improve nest success. However, the risks and benefits of different actions were uncertain; for example, protecting nests from predators might also increase the risk of nest desertion by adults. To avoid risks, the recovery team opted to collect more information before implementation. The two methods of nest protection were compared using a field experiment. However, the same risk aversion limited the experiment to a single variable (nest predation) and dictated the use of artificial nests. The results of the experiment suggested neither action was likely to significantly reduce predation risks (<3% mean differences in survival between treatment and control). When presented with these results, managers made only minor revisions to their estimates; in part, this reflected low confidence by managers that artificial nests could reflect real predation risks. However, estimates were also revised more negatively for the initially less-favored option, despite absence of such evidence, possibly highlighting confirmation bias. In this uncertain situation, the status quo was initially maintained although it was perceived as suboptimal; implementation of the preferred option (tree collars) is now planned for the 2019 breeding season. We faced what might be a common conundrum for conservation of critically endangered species. High uncertainty affects management decisions; however, perilous species status also leads to strong risk aversion, which limits both the willingness to act on limited information and the ability to learn effectively. Structured methods can increase transparency, facilitate evaluation, and assist decision making, but objective limitations and subjective attitudes cannot be circumvented entirely.

show abstract

Section: Discussionmentioning

confidence: 99%

Risk aversion and uncertainty create a conundrum for planning recovery of a critically endangered species

Canessa

Taylor

Clarke

et al. 2019

Conservat Sci and Prac

View full text Add to dashboard Cite

show abstract

“…The marked effect of information uncertainty on achievement of conservation objectives highlights the importance of improving accuracy of species response estimates to management. This may require to refine the expert elicitation process by, for instance, giving experts opportunities to discuss their answers in person and testing the experts beforehand with similar questions to the ones they will be required to answer (Burgman et al., ; Hemming, Burgman, Hanea, McBride, & Wintle, ). However, insufficient resources (time and money) often require managers to make quick decisions with the best available information, that is, experts' best guesses.…”

Section: Discussionmentioning

confidence: 99%

Information uncertainty influences conservation outcomes when prioritizing multi‐action management efforts

Cattarino

Hermoso

Carwardine

et al. 2018

Journal of Applied Ecology

View full text Add to dashboard Cite

In managing various threats to biodiversity, it is important to prioritize multiple management actions and the levels of effort to apply. However, a spatial conservation prioritization framework that integrates these key aspects, and can be generalized, is still missing. Moreover, assessing the robustness of prioritization frameworks to uncertainty in species responses to management is critical to avoid misallocation of limited resources. Yet, the impact of information uncertainty on prioritization of management effort remains unknown. We present an approach for prioritizing alternative levels of conservation management effort to multiple actions, based on the ecological responses of species to management. We estimated species responses through a structured email‐based expert elicitation process, where we also captured the uncertainty in individual experts' assessments. We identified priority locations and associated level of management of effort of four actions to abate threats to freshwater‐dependent fauna, using a northern Australia case study, and quantified sensitivity of the proposed solution to uncertainty in the answers of each individual expert. Achievement of conservation targets for freshwater‐dependent fauna in the Daly River catchment would require 9.4 million AU$ per year, for a total of approximately 189 million AU$ investment over 20 years. We suggest that this could be best achieved through a mix of aerial shooting of buffalos and pigs, riparian fencing and chemical spraying of weeds, applied at varying levels of management effort in key areas of the catchment. Uncertainty in experts' estimation of species responses to threats causes 60% of the species to achieve 80% of their conservation targets, which was consistent across target levels. Synthesis and applications. Our prioritization approach facilitates the planning of conservation management at fine spatial scales and is applicable to terrestrial, freshwater and marine realms. Plan implementation may require policy instruments ranging from landowner stewardship agreements, market‐based mechanisms and low‐intensity land use management schemes, to regulation of commercial activities within portions of marine protected areas. However, assessing plan sensitivity to uncertainty in species response to management and finding ways of dealing with it in the prioritization rather than ignoring it, as often done, remains vital for effective achievement of conservation objectives.

show abstract

“…The IDEA protocol places an emphasis on recruiting a diversity of individuals because, whereas expertise for judgements under uncertainty cannot be easily predicted a priori, diverse groups with knowledge of the questions often form well‐calibrated and accurate judgements …”

Section: Methodsmentioning

confidence: 99%

“…IDEA stands for key steps, “Investigate,” “Discuss,” “Estimate,” and “Aggregate.” Prescriptive advice for the protocol is outlined in Hemming et al; however, briefly, it involves the following steps: A diverse group of experts is recruited to answer questions with probabilistic or quantitative responses. The experts are asked to first Investigate the questions and to clarify their meanings and then to provide their private, individual estimates with uncertainty. The experts receive feedback on their estimates in relation to other experts. With assistance of a facilitator, the experts are encouraged to Discuss the results, resolve different interpretations of the questions, cross‐examine reasoning and evidence, and then provide a second and final private Estimate . The individual estimates are then combined using mathematical Aggregation , usually an equally weighted aggregation. …”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Improving expert forecasts in reliability: Application and evidence for structured elicitation protocols

Hemming

Armstrong

Burgman

et al. 2019

Quality & Reliability Eng

Self Cite

View full text Add to dashboard Cite

Quantitative expert judgements are used in reliability assessments to inform critically important decisions. Structured elicitation protocols have been advocated to improve expert judgements, yet their application in reliability is challenged by a lack of examples or evidence that they improve judgements. This paper aims to overcome these barriers. We present a case study where two world‐leading protocols, the IDEA protocol and the Classical Model, were combined and applied by the Australian Department of Defence for a reliability assessment. We assess the practicality of the methods and the extent to which they improve judgements. The average expert was extremely overconfident, with 90% credible intervals containing the true realisation 36% of the time. However, steps contained in the protocols substantially improved judgements. In particular, an equal weighted aggregation of individual judgements and the inclusion of a discussion phase and revised estimate helped to improve calibration, statistical accuracy, and the Classical Model score. Further improvements in precision and information were made via performance weighted aggregation. This paper provides useful insights into the application of structured elicitation protocols for reliability and the extent to which judgements are improved. The findings raise concerns about existing practices for utilising experts in reliability assessments and suggest greater adoption of structured protocols is warranted. We encourage the reliability community to further develop examples and insights.

show abstract

A practical guide to structured expert elicitation using the IDEA protocol

Cited by 301 publications

References 62 publications

Risk aversion and uncertainty create a conundrum for planning recovery of a critically endangered species

Risk aversion and uncertainty create a conundrum for planning recovery of a critically endangered species

Information uncertainty influences conservation outcomes when prioritizing multi‐action management efforts

Improving expert forecasts in reliability: Application and evidence for structured elicitation protocols

Contact Info

Product

Resources

About