Expert judgement informs a variety of important applications in conservation andnatural resource management, including threatened species management, environmental impact assessment and structured decision-making. However, expert judgements can be prone to contextual biases. Structured elicitation protocols mitigate these biases, and improve the accuracy and transparency of the resulting judgements. Despite this, the elicitation of expert judgement within conservation and natural resource management remains largely informal. We suggest this may be attributed to financial and practical constraints, which are not addressed by many existing structured elicitation protocols.2. In this paper, we advocate that structured elicitation protocols must be adopted when expert judgements are used to inform science. In order to motivate a wider adoption of structured elicitation protocols, we outline the IDEA protocol. The protocol improves the accuracy of expert judgements and includes several key steps which may be familiar to many conservation researchers, such as the four-step elicitation, and a modified Delphi procedure ("Investigate," "Discuss," "Estimate" and "Aggregate"). It can also incorporate remote elicitation, making structured expert judgement accessible on a modest budget.3. The IDEA protocol has recently been outlined in the scientific literature; however, a detailed description has been missing. This paper fills that important gap by clearly outlining each of the steps required to prepare for and undertake an elicitation. 4. While this paper focuses on the need for the IDEA protocol within conservation and natural resource management, the protocol (and the advice contained in this paper) is applicable to a broad range of scientific domains, as evidenced by its application to biosecurity, engineering and political forecasting. By clearly outlining the IDEA protocol, we hope that structured protocols will be more widely understood and adopted, resulting in improved judgements and increased transparency when expert judgement is required. K E Y W O R D SDelphi, expert elicitation, forecasting, four-step elicitation, IDEA protocol, quantitative estimates, structured expert judgement

IntroductionNatural resource management uses expert judgement to estimate facts that inform important decisions. Unfortunately, expert judgement is often derived by informal and largely untested protocols, despite evidence that the quality of judgements can be improved with structured approaches. We attribute the lack of uptake of structured protocols to the dearth of illustrative examples that demonstrate how they can be applied within pressing time and resource constraints, while also improving judgements.Aims and methodsIn this paper, we demonstrate how the IDEA protocol for structured expert elicitation may be deployed to overcome operational challenges while improving the quality of judgements. The protocol was applied to the estimation of 14 future abiotic and biotic events on the Great Barrier Reef, Australia. Seventy-six participants with varying levels of expertise related to the Great Barrier Reef were recruited and allocated randomly to eight groups. Each participant provided their judgements using the four-step question format of the IDEA protocol (‘Investigate’, ‘Discuss’, ‘Estimate’, ‘Aggregate’) through remote elicitation. When the events were realised, the participant judgements were scored in terms of accuracy, calibration and informativeness.Results and conclusionsThe results demonstrate that the IDEA protocol provides a practical, cost-effective, and repeatable approach to the elicitation of quantitative estimates and uncertainty via remote elicitation. We emphasise that i) the aggregation of diverse individual judgements into pooled group judgments almost always outperformed individuals, and ii) use of a modified Delphi approach helped to remove linguistic ambiguity, and further improved individual and group judgements. Importantly, the protocol encourages review, critical appraisal and replication, each of which is required if judgements are to be used in place of data in a scientific context. The results add to the growing body of literature that demonstrates the merit of using structured elicitation protocols. We urge decision-makers and analysts to use insights and examples to improve the evidence base of expert judgement in natural resource management.

No abstract

No abstract

No abstract

Bayesian belief nets (BBNs) have become a popular tool for specifying highdimensional probabilistic models. Commercial tools with an advanced graphical user interface that support BBNs construction and inference are available. Thus, building and working with BBNs is very efficient as long as one is not forced to quantify complex BBNs. A high assessment burden of discrete BBNs is often caused by the discretization of continuous variables. Until recently, continuous BBNs were restricted to the joint normal distribution. We present the 'copula-vine' approach to continuous BBNs. This approach is quite general and allows traceable and defendable quantification methods, but it comes at a price: these BBNs must be evaluated by Monte Carlo simulation. Updating such a BBN requires re-sampling the whole structure. The advantages of fast updating algorithms for discrete BBNs are decisive. A hybrid method advanced here samples the continuous BBN once, and then discretizes this so as to enable fast updating. This combines the reduced assessment burden and modelling flexibility of the continuous BBNs with the fast updating algorithms of discrete BBNs. Sampling large complex structures only once can still involve time consuming numerical calculations. Therefore a new sampling protocol based on normal vines is developed. Normal vines are used to realize the dependence structure specified via (conditional) rank correlations on the continuous BBN. We will emphasize the advantages of this method by means of examples.

Many applications in decision making under uncertainty and probabilistic risk assessment require the assessment of multiple, dependent uncertain quantities, so that in addition to marginal distributions, interdependence needs to be modelled in order to properly understand the overall risk. Nevertheless, relevant historical data on dependence information are often not available or simply too costly to obtain. In this case, the only sensible option is to elicit this uncertainty through the use of expert judgements. In expert judgement studies, a structured approach to eliciting variables of interest is desirable so that their assessment is methodologically robust. One of the key decisions during the elicitation process is the form in which the uncertainties are elicited. This choice is subject to various, potentially conflicting, desiderata related to e.g. modelling convenience, coherence between elicitation parameters and the model, combining judgements, and the assessment burden for the experts. While extensive and systematic guidance to address these considerations exists for single variable uncertainty elicitation, for higher dimensions very little such guidance is available. Therefore this paper offers a systematic review of the current literature on eliciting dependence. The literature on the elicitation of dependence parameters such as correlations is presented alongside commonly used dependence models and experience from case studies. From this, guidance about the strategy for dependence assessment is given and gaps in the existing research are identified to determine future directions for structured methods to elicit dependence. (Anca M. Hanea), O.MoralesNapoles@tudelft.nl (Oswaldo Morales-Nápoles) cause variables in the model are correlated, or indirectly when an uncertainty analysis of model parameters is carried out to explore model robustness. Both cases exhibit complex interrelations and dependencies which need to be considered if assumptions such as independence are not 10 justifiable.However, it is often not straightforward to either model or quantify dependence. In particular whenever no relevant historical data are available, the only sensible way to achieve uncertainty quantification is through eliciting ex-15

Expert judgement is pervasive in all forms of risk analysis, yet the development of tools to deal with such judgements in a repeatable and transparent fashion is relatively recent. This work outlines new findings related to an approach to expert elicitation termed the IDEA protocol. IDEA combines psychologically robust interactions among experts with mathematical aggregation of individual estimates. In particular, this research explores whether communication among experts adversely effects the reliability of group estimates. Using data from estimates of the outcomes of geopolitical events, we find that loss of independence is relatively modest and it is compensated by improvements in group accuracy.