Weighting and aggregating expert ecological judgments

Abstract

Citation: Hemming, V., A. M. Hanea, T. Walshe, and M. A. Burgman. 2020. Weighting and aggregating expert ecological judgments. Ecological Applications 30(4):Abstract. Performance weighted aggregation of expert judgments, using calibration questions, has been advocated to improve pooled quantitative judgments for ecological questions. However, there is little discussion or practical advice in the ecological literature regarding the application, advantages or challenges of performance weighting. In this paper we (1) illustrate how the IDEA protocol with four-step question format can be extended to include performance weighted aggregation from the Classical Model, and (2) explore the extent to which this extension improves pooled judgments for a range of performance measures. Our case study demonstrates that performance weights can improve judgments derived from the IDEA protocol with four-step question format. However, there is no a-priori guarantee of improvement. We conclude that the merits of the method lie in demonstrating that the final aggregation of judgments provides the best representation of uncertainty (i.e., validation), whether that be via equally weighted or performance weighted aggregation. Whether the time and effort entailed in performance weights can be justified is a matter for decision-makers. Our case study outlines the rationale, challenges, and benefits of performance weighted aggregations. It will help to inform decisions about the deployment of performance weighting and avoid common pitfalls in its application.

Red hot frogs: identifying the Australian frogs most at risk of extinction

“…However, this approach is likely to require significantly more time and effort, comes with several additional challenges (e.g. developing questions about future events for which data can be obtained within a reasonable timeframe for validating participant performance), and there is no guarantee that it will improve estimates (Hemming et al 2020). We consider the approach we used here provides an effective means for assessing extinction risk relative to selecting one or a few seemingly well-credentialed experts.…”

Using expert knowledge to support Endangered Species Act decision‐making for data‐deficient species

“…Other researchers have used calibration variables designed to test experts' statistical accuracy on quantities that will be known in the near future (Wittmann et al 2015). However, this approach is rare in ecological applications because of the additional time required of the experts and the difficulty of identifying relevant test quantities (Hemming et al 2020). Although quantities such as annual biomass of commercial fishes may provide useful seed questions for some management problems (Wittmann et al 2015), it is difficult to identify questions that reasonably test an expert's ability to judge population or species persistence over several generations.…”

“…However, this approach is rare in ecological applications because of the additional time required of the experts and the difficulty of identifying relevant test quantities (Hemming et al. 2020 ). Although quantities such as annual biomass of commercial fishes may provide useful seed questions for some management problems (Wittmann et al.…”

Predicting reliability through structured expert elicitation with repliCATS (Collaborative Assessments for Trustworthy Science)

“…These help to improve accuracy, and reduce overconfidence by reducing biases such as anchoring, groupthink, and confirmation bias. The IDEA protocol aims to improve accuracy through controlling groupthink by aggregating group assessments mathematically and not behaviourally [12][13][14][15]. That is, group members are not forced to agree on a single final judgement that reflects the whole group.…”