Augmenting Expert Elicitation with Structured Visual Deliberation

Gelder, Tim van; Vodicka, Roger; Armstrong, Nicholas

doi:10.1002/app5.145

Cited by 5 publications

(15 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While this paper emphasises the suitability of the approach in conservation and natural resource management, it should be noted that the IDEA protocol is equally suited to a wide variety of scientific and technical domains. This is evidenced by its effective application in geopolitical forecasting (Hanea, McBride, Burgman, & Wintle, ; Hanea et al., ; Wintle et al., ) and engineering (van Gelder, Vodicka, & Armstrong, ).…”

Section: Introductionmentioning

confidence: 99%

A practical guide to structured expert elicitation using the IDEA protocol

et al. 2017

View full text Add to dashboard Cite

Expert judgement informs a variety of important applications in conservation andnatural resource management, including threatened species management, environmental impact assessment and structured decision-making. However, expert judgements can be prone to contextual biases. Structured elicitation protocols mitigate these biases, and improve the accuracy and transparency of the resulting judgements. Despite this, the elicitation of expert judgement within conservation and natural resource management remains largely informal. We suggest this may be attributed to financial and practical constraints, which are not addressed by many existing structured elicitation protocols.2. In this paper, we advocate that structured elicitation protocols must be adopted when expert judgements are used to inform science. In order to motivate a wider adoption of structured elicitation protocols, we outline the IDEA protocol. The protocol improves the accuracy of expert judgements and includes several key steps which may be familiar to many conservation researchers, such as the four-step elicitation, and a modified Delphi procedure ("Investigate," "Discuss," "Estimate" and "Aggregate"). It can also incorporate remote elicitation, making structured expert judgement accessible on a modest budget.3. The IDEA protocol has recently been outlined in the scientific literature; however, a detailed description has been missing. This paper fills that important gap by clearly outlining each of the steps required to prepare for and undertake an elicitation. 4. While this paper focuses on the need for the IDEA protocol within conservation and natural resource management, the protocol (and the advice contained in this paper) is applicable to a broad range of scientific domains, as evidenced by its application to biosecurity, engineering and political forecasting. By clearly outlining the IDEA protocol, we hope that structured protocols will be more widely understood and adopted, resulting in improved judgements and increased transparency when expert judgement is required. K E Y W O R D SDelphi, expert elicitation, forecasting, four-step elicitation, IDEA protocol, quantitative estimates, structured expert judgement

show abstract

Section: Introductionmentioning

confidence: 99%

A practical guide to structured expert elicitation using the IDEA protocol

et al. 2017

View full text Add to dashboard Cite

show abstract

“…The same general format of the questions of interest that had been devised by van Gelder et al was used:…”

Section: Methodsmentioning

confidence: 99%

“…They adopted the IDEA protocol to elicit judgements from experts, which could be used to adjust parameters in the model. [28][29][30]32 In their study on the Skua, van Gelder et al 37 found that the IDEA protocol was practical to apply and improved the transparency and the information obtained to inform decisions. There were, however, aspects of the elicitation that van Gelder et al 37 believed could be improved.…”

Section: Motivation For This Studymentioning

confidence: 99%

“…In 2015, the Australian Department of Defence began investigating how structured elicitation protocols could be used to improve expert judgements. At that time, the expected attrition rates for a proposed aircraft procurement termed the “Skua” (a pseudonym) were required . As noted by van Gelder et al, the Australian Defence Force (ADF) had never flown these vehicles.…”

Section: Introductionmentioning

confidence: 99%

“…In their study on the Skua, van Gelder et al found that the IDEA protocol was practical to apply and improved the transparency and the information obtained to inform decisions. There were, however, aspects of the elicitation that van Gelder et al believed could be improved.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Improving expert forecasts in reliability: Application and evidence for structured elicitation protocols

Hemming

Armstrong

Burgman

et al. 2019

Quality & Reliability Eng

Self Cite

View full text Add to dashboard Cite

Quantitative expert judgements are used in reliability assessments to inform critically important decisions. Structured elicitation protocols have been advocated to improve expert judgements, yet their application in reliability is challenged by a lack of examples or evidence that they improve judgements. This paper aims to overcome these barriers. We present a case study where two world‐leading protocols, the IDEA protocol and the Classical Model, were combined and applied by the Australian Department of Defence for a reliability assessment. We assess the practicality of the methods and the extent to which they improve judgements. The average expert was extremely overconfident, with 90% credible intervals containing the true realisation 36% of the time. However, steps contained in the protocols substantially improved judgements. In particular, an equal weighted aggregation of individual judgements and the inclusion of a discussion phase and revised estimate helped to improve calibration, statistical accuracy, and the Classical Model score. Further improvements in precision and information were made via performance weighted aggregation. This paper provides useful insights into the application of structured elicitation protocols for reliability and the extent to which judgements are improved. The findings raise concerns about existing practices for utilising experts in reliability assessments and suggest greater adoption of structured protocols is warranted. We encourage the reliability community to further develop examples and insights.

show abstract

Building less-flawed metrics: Understanding and creating better measurement and incentive systems

Manheim

2023

Patterns

View full text Add to dashboard Cite

Augmenting Expert Elicitation with Structured Visual Deliberation

Cited by 5 publications

References 19 publications

A practical guide to structured expert elicitation using the IDEA protocol

A practical guide to structured expert elicitation using the IDEA protocol

Improving expert forecasts in reliability: Application and evidence for structured elicitation protocols

Building less-flawed metrics: Understanding and creating better measurement and incentive systems

Contact Info

Product

Resources

About