Comparison of three expert elicitation methods for logistic regression on predicting the presence of the threatened brush‐tailed rock‐wallaby Petrogale penicillata

Abstract

SUMMARYNumerous expert elicitation methods have been suggested for generalised linear models (GLMs). This paper compares three relatively new approaches to eliciting expert knowledge in a form suitable for Bayesian logistic regression. These methods were trialled on two experts in order to model the habitat suitability of the threatened Australian brush-tailed rock-wallaby (Petrogale penicillata). The first elicitation approach is a geographically assisted indirect predictive method with a geographic information system (GIS) interface. The second approach is a predictive indirect method which uses an interactive graphical tool. The third method uses a questionnaire to elicit expert knowledge directly about the impact of a habitat variable on the response. Two variables (slope and aspect) are used to examine prior and posterior distributions of the three methods. The results indicate that there are some similarities and dissimilarities between the expert informed priors of the two experts formulated from the different approaches. The choice of elicitation method depends on the statistical knowledge of the expert, their mapping skills, time constraints, accessibility to experts and funding available. This trial reveals that expert knowledge can be important when modelling rare event data, such as threatened species, because experts can provide additional information that may not be represented in the dataset. However care must be taken with the way in which this information is elicited and formulated.

Methods to Elicit Probability Distributions from Experts: A Systematic Review of Reported Practice in Health Technology Assessment

“…However, the perceived complexity of expert elicitation may limit its use. Trade-offs are inherent to the elicitation process, as assessors need to balance limited resources and deadlines with achieving optimal response from the experts [35]. In the included studies, most choices were made with investigators reportedly being aware of potential trade-offs.…”

Essays on probability elicitation scoring rules

“…The purpose is to evaluate the performance of p in modeling the likelihood of rare events. Rare events have been of interest for handling problems from diverse contexts, such as natural catastrophes 18 and risk assessment 19,20 , and therefore have been frequently the focus of elicitation exercises 21 . Concerning this characteristic, on can see that no extension of the evidence-based SRs perform well; they do not provide a relative weight of the discrepancy between p j and g j .…”

What is an expert? A systems perspective on expertise

“…Consequently, expert knowledge is being increasingly used in a diverse range of disciplines where more traditional types of empirical data are insufficient to address particular issues in a specific context and/or in a timely manner. These discipline areas include landscape ecology ), conservation and management of threatened and endangered species (Campbell 2002;Smith et al 2007;Murray et al 2009;O'Leary et al 2009;James et al 2010;Johnson et al 2010;Martin et al 2012), environmental risk (Hamilton et al 2007;Hoelzer et al 2012;Johnson et al 2013a,b), meteorology (Risk Management Services 2006), climate change (Risbey 2008), health and medicine (Knol et al 2010;Waterhouse and Johnson 2012), knowledge engineering (Kendal and Creen 2007), information technology systems (Franke et al 2012) and industry (Yu 2002). Central to the use of expert knowledge in these situations are the subjective probabilities associated with the elicitation of expert knowledge (Cox 2000;O'Hagan et al 2006).…”

“…Because of the limits to expert knowledge, it is common to elicit multiple experts possessing expertise in a range of disciplines, and multiple experts in the same disciplines, when addressing a particular issue (Martin et al 2005). In such situations, expert judgements can be expected to vary (e.g., Campbell 2002;Martin et al 2005;O'Leary et al 2009) and combining and/or weighting (Burgman et al 2011) opinions of multiple experts should provide better aggregate judgements. However, in some circumstances, such as policy decisions, it may be preferable to represent the diversity of expert judgements for effective and informed decision-making and planning, rather than presenting an aggregated unified position.…”