Elicitation of Quantitative Data from a Heterogeneous Expert Panel: Formal Process and Application in Animal Health

Abstract

This paper presents a protocol for a formal expert judgment process using a heterogeneous expert panel aimed at the quantification of continuous variables. The emphasis is on the process's requirements related to the nature of expertise within the panel, in particular the heterogeneity of both substantive and normative expertise. The process provides the opportunity for interaction among the experts so that they fully understand and agree upon the problem at hand, including qualitative aspects relevant to the variables of interest, prior to the actual quantification task. Individual experts' assessments on the variables of interest, cast in the form of subjective probability density functions, are elicited with a minimal demand for normative expertise. The individual experts' assessments are aggregated into a single probability density function per variable, thereby weighting the experts according to their expertise. Elicitation techniques proposed include the Delphi technique for the qualitative assessment task and the ELI method for the actual quantitative assessment task. Appropriately, the Classical model was used to weight the experts' assessments in order to construct a single distribution per variable. Applying this model, the experts' quality typically was based on their performance on seed variables. An application of the proposed protocol in the broad and multidisciplinary field of animal health is presented. Results of this expert judgment process showed that the proposed protocol in combination with the proposed elicitation and analysis techniques resulted in valid data on the (continuous) variables of interest. In conclusion, the proposed protocol for a formal expert judgment process aimed at the elicitation of quantitative data from a heterogeneous expert panel provided satisfactory results. Hence, this protocol might be useful for expert judgment studies in other broad and/or multidisciplinary fields of interest.

Methods to elicit beliefs for Bayesian priors: a systematic review

“…Investigators had asked participants about the mean [32,33,40], median [27,30,45] and mode [16,20,37] for a parameter. Participants had been asked to estimate the probability of an outcome/ event [14,15,19,24,28,42e44,46], the proportion of individuals who will have an outcome [3,17,47], the relative risk of an outcome [35,36], the value for a dependent variable given specified values for independent variables [22,23], and their weight of belief [5,38,39,41].…”

“…Investigators have asked participants about measures of central tendency [16,20,27,30,32,33,37,40,45], probability [14,15,19,24,28,42e44,46], proportion [3,17,47], relative risk [35,36], value for a dependent variable given specified values for independent variables [22,23], and their weight of belief [5,38,39,41]. Insufficient normative goodness (statistical understanding) and insufficient understanding of the elicitation question threaten the validity of the belief elicited [44].…”

Diseases of livestock in the Pacific Islands region: Setting priorities for food animal biosecurity

“…It appears that only a limited number of prioritization exercises have been implemented for animal diseases globally (Humblet et al, 2012;McKenzie et al, 2007;Phylum, 2009;Van der Fels-Klerx et al, 2002). In the Pacific Islands region, a semi-quantitative prioritization process has been conducted by the public health sector of the Federate States of Micronesia for a revised selection of diseases to include in the National Notifiable Diseases List (Pavlin et al, 2010).…”

Critical review of methodology and application of risk ranking for prioritisation of food and feed related issues, on the basis of the size of anticipated health impact

“…participants possessing expert knowledge and experience (Van der Fels-Klerx et al, 2002). It provides feedback of the results and comments from previous rounds, thereby simulating some of the interaction of face-to-face meetings.…”