Eliciting improved quantitative judgements using the IDEA protocol: A case study in natural resource management

Abstract

IntroductionNatural resource management uses expert judgement to estimate facts that inform important decisions. Unfortunately, expert judgement is often derived by informal and largely untested protocols, despite evidence that the quality of judgements can be improved with structured approaches. We attribute the lack of uptake of structured protocols to the dearth of illustrative examples that demonstrate how they can be applied within pressing time and resource constraints, while also improving judgements.Aims and methodsIn this paper, we demonstrate how the IDEA protocol for structured expert elicitation may be deployed to overcome operational challenges while improving the quality of judgements. The protocol was applied to the estimation of 14 future abiotic and biotic events on the Great Barrier Reef, Australia. Seventy-six participants with varying levels of expertise related to the Great Barrier Reef were recruited and allocated randomly to eight groups. Each participant provided their judgements using the four-step question format of the IDEA protocol (‘Investigate’, ‘Discuss’, ‘Estimate’, ‘Aggregate’) through remote elicitation. When the events were realised, the participant judgements were scored in terms of accuracy, calibration and informativeness.Results and conclusionsThe results demonstrate that the IDEA protocol provides a practical, cost-effective, and repeatable approach to the elicitation of quantitative estimates and uncertainty via remote elicitation. We emphasise that i) the aggregation of diverse individual judgements into pooled group judgments almost always outperformed individuals, and ii) use of a modified Delphi approach helped to remove linguistic ambiguity, and further improved individual and group judgements. Importantly, the protocol encourages review, critical appraisal and replication, each of which is required if judgements are to be used in place of data in a scientific context. The results add to the growing body of literature that demonstrates the merit of using structured elicitation protocols. We urge decision-makers and analysts to use insights and examples to improve the evidence base of expert judgement in natural resource management.

Improving expert forecasts in reliability: Application and evidence for structured elicitation protocols

“…While our sample size is small, and the best performing expert in Round 1 did not supply demographic data, there appears to be no overall trend in performance associated with years of experience or self‐rating. These findings reflect a suite of studies that have failed to find a connection between good judgements under uncertainty and credentials …”

“…Following Round 1, the IDEA protocol involves a subsequent discussion phase and an opportunity for experts to revise their judgement (Round 2). The potential advantages of discussion and whether it improves or degrades the quality of judgements have been debated …”

“…This step reduces overconfidence relative to eliciting fixed intervals . Its application across a range of domains, and via remote elicitation (where a facilitator is not present), suggests it is helpful for overcoming the persistent challenge faced in deriving quantitative judgements from experts …”

“…Applying these steps has been shown to substantially improve the accuracy of point estimates and the calibration of interval judgements . Such judgements can be obtained from relatively small groups of five to nine experts using face‐to‐face workshops or remote elicitation …”

“…The inclusion of an initial round of estimation, followed by a discussion, and a revised estimate means IDEA can be likened to a Delphi style elicitation . Delphi is not well defined and encompasses a large array of approaches .…”