Structured elicitation of expert judgments for threatened species assessment: a case study on a continental scale using email

Abstract

Summary1. Expert knowledge is used routinely to inform listing decisions under the IUCN Red List criteria. Differences in opinion arise between experts in the presence of epistemic uncertainty, as a result of different interpretations of incomplete information and differences in individual beliefs, values and experiences. Structured expert elicitation aims to anticipate and account for such differences to increase the accuracy of final estimates. 2. A diverse panel of 16 experts independently evaluated up to 125 parameters per taxon to assess the IUCN Red List category of extinction risk for nine Australian bird taxa. Each panellist was provided with the same baseline data. Additional judgments and advice were sought from taxon specialists outside the panel. One question set elicited lowest and highest plausible estimates, best estimates and probabilities that the true values were contained within the upper and lower bounds. A second question set elicited yes ⁄ no answers and a degree of credibility in the answer provided. 3. Once initial estimates were obtained, all panellists were shown each others' values. They discussed differences and reassessed their original values. Most communication was carried out by email. 4. The process took nearly 6 months overall to complete, and required an average of 1 h and up to 13 h per taxon for a panellist to complete the initial assessment. 5. Panellists were mostly in agreement with one another about IUCN categorisations for each taxon. Where they differed, there was some evidence of convergence in the second round of assessments, although there was persistent non-overlap for about 2% of estimates. The method exposed evidence of common subjective biases including overconfidence, anchoring to available data, definitional ambiguity and the conceptual difficulty of estimating percentages rather than natural numbers. 6. This study demonstrates the value of structured elicitation techniques to identify and to reduce potential sources of bias and error among experts. The formal nature of the process meant that the consensus position reached carried greater weight in subsequent deliberations on status. The structured process is worthwhile for high profile or contentious taxa, but may be too time intensive for less divisive cases.

Expert elicitation of seasonal abundance of North Atlantic right whales Eubalaena glacialis in the mid-Atlantic

“…Similar to McBride et al (2012), we used a 4-part elicitation procedure (Speirs-Bridge et al 2010) and allowed panel members to revise their answers after feedback and group discussion. McBride et al (2012) used the percentage confidence provided by the experts with respect to each elicited value as confidence intervals, and used linear extrapolation to standardize the confidence intervals to 100% credible intervals around the best estimate. From these standardized intervals, McBride et al (2012) used the Delphi method to obtain the mean of the group's best estimate of each value and the means of the normalized upper and lower bounds of the confidence intervals.…”

“…McBride et al (2012) used the percentage confidence provided by the experts with respect to each elicited value as confidence intervals, and used linear extrapolation to standardize the confidence intervals to 100% credible intervals around the best estimate. From these standardized intervals, McBride et al (2012) used the Delphi method to obtain the mean of the group's best estimate of each value and the means of the normalized upper and lower bounds of the confidence intervals. In contrast, we used the percentage confidence to weight each expert's estimate when merging distributions for the parameters.…”

Protected area management under climate change: A framework for decision making

“…For the individual interviews, group averages and standard deviations were calculated from the initial estimates. These were then made available to the experts, who then had the option of adjusting their original estimates (Linstone & Turoff 1975;Martin et al 89 2012;McBride et al 2012). Final averages were used for the conditional probability tables in the BBNs .…”

Modelling seasonal habitat suitability and connectivity for feral pigs in northern Australia: towards risk-based management of infectious animal diseases with wildlife hosts

“…Consequently, a number of recent publications have argued that expert knowledge must not be perceived as a "cheap, readily available source of knowledge" (McBride 2013, p. 156) and advocated a structured approach that treats expert elicitation as a form of scientific data collection (Kuhnert et al 2010;Johnson et al 2012b;McBride et al 2012a). Like all data, expert knowledge has limits: it is subjective, incomplete and often unstructured.…”

“…Method B restricted elicitation to key scenarios as in Cain (2001). However, instead of directly assigning probabilities to each state of the response variable, we used interval judgements (Speirs-Bridge et al 2010;McBride et al 2012a), asking experts for their best estimate and the outer bounds of a 95% confidence interval. To maintain consistency with method A, we allowed only TNormal distributions centred on the best estimate.…”