Experience with Expert Judgment: the<scp>TU</scp>Delft Expert Judgment Data

Cooke, Roger M.; Goossens, Louis L.H.J.

doi:10.1002/9780470087923.hhs014

Cited by 15 publications

(39 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most importantly, only four experts participated in the final elicitation procedure. Four experts can be sufficient, but a panel of about eight is recommended (Cooke and Goossens, 1999 ). When more experts are involved, it is easier to recognize the general opinion and the final result is less sensitive to the misjudgment of one expert.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Application and Evaluation of an Expert Judgment Elicitation Procedure for Correlations

Zondervan‐Zwijnenburg

Schoot-Hubeek²,

Lek

et al. 2017

Front. Psychol.

View full text Add to dashboard Cite

The purpose of the current study was to apply and evaluate a procedure to elicit expert judgments about correlations, and to update this information with empirical data. The result is a face-to-face group elicitation procedure with as its central element a trial roulette question that elicits experts' judgments expressed as distributions. During the elicitation procedure, a concordance probability question was used to provide feedback to the experts on their judgments. We evaluated the elicitation procedure in terms of validity and reliability by means of an application with a small sample of experts. Validity means that the elicited distributions accurately represent the experts' judgments. Reliability concerns the consistency of the elicited judgments over time. Four behavioral scientists provided their judgments with respect to the correlation between cognitive potential and academic performance for two separate populations enrolled at a specific school in the Netherlands that provides special education to youth with severe behavioral problems: youth with autism spectrum disorder (ASD), and youth with diagnoses other than ASD. Measures of face-validity, feasibility, convergent validity, coherence, and intra-rater reliability showed promising results. Furthermore, the current study illustrates the use of the elicitation procedure and elicited distributions in a social science application. The elicited distributions were used as a prior for the correlation, and updated with data for both populations collected at the school of interest. The current study shows that the newly developed elicitation procedure combining the trial roulette method with the elicitation of correlations is a promising tool, and that the results of the procedure are useful as prior information in a Bayesian analysis.

show abstract

Section: Discussionmentioning

confidence: 99%

“…Morales-Nápoles ( 2010 ) used this method to elicit a distribution for a correlation. After the elicitation phase, distributions are fitted to the elicited quantiles (Cooke and Goossens, 1999 ).…”

Section: Introductionmentioning

confidence: 99%

Application and Evaluation of an Expert Judgment Elicitation Procedure for Correlations

Zondervan‐Zwijnenburg

Schoot-Hubeek²,

Lek

et al. 2017

Front. Psychol.

View full text Add to dashboard Cite

show abstract

“…However, it is doubtful whether such estimates provide a good indication of the actual value of the elicited information, or merely introduce more bias [11,85]. …”

Section: Seven Step Procedures For Formal Expert Elicitationmentioning

confidence: 99%

The use of expert elicitation in environmental health impact assessment: a seven step procedure

et al. 2010

View full text Add to dashboard Cite

BackgroundEnvironmental health impact assessments often have to deal with substantial uncertainties. Typically, the knowledge-base is limited with incomplete, or inconsistent evidence and missing or ambiguous data. Consulting experts can help to identify and address uncertainties.MethodsFormal expert elicitation is a structured approach to systematically consult experts on uncertain issues. It is most often used to quantify ranges for poorly known parameters, but may also be useful to further develop qualitative issues such as definitions, assumptions or conceptual (causal) models. A thorough preparation and systematic design and execution of an expert elicitation process may increase the validity of its outcomes and transparency and trustworthiness of its conclusions. Various expert elicitation protocols and methods exist. However, these are often not universally applicable, and need customization to suite the needs of a specific study. In this paper, we set out to develop a widely applicable method for the use of expert elicitation in environmental health impact assessment.ResultsWe present a practical yet flexible seven step procedure towards organising expert elicitation in the context of environmental health impact assessment, based on existing protocols. We describe how customization for specific applications is always necessary. In particular, three issues affect the choice of methods for a particular application: the types of uncertainties considered, the intended use of the elicited information, and the available resources. We outline how these three considerations guide choices regarding the design and execution of expert elicitation. We present signposts to sources where the issues are discussed in more depth to give the newcomer the insights needed to make the protocol work. The seven step procedure is illustrated using examples from earlier published elicitations in the field of environmental health research.ConclusionsWe conclude that, despite some known criticism on its validity, formal expert elicitation can support environmental health research in various ways. Its main purpose is to provide a temporary summary of the limited available knowledge, which can serve as a provisional basis for policy until further research has been carried out.

show abstract

“… Cooke and Goossens (1999) describe that a panel of four experts can be sufficient for an elicitation, but they recommend a panel of about eight experts as a rule of thumb. In the user feasibility study nine respondents participated.…”

Section: Elicitation Studiesmentioning

confidence: 99%

Proposal for a Five-Step Method to Elicit Expert Judgment

Veen

Stoel²,

Zondervan‐Zwijnenburg

et al. 2017

Front. Psychol.

View full text Add to dashboard Cite

Elicitation is a commonly used tool to extract viable information from experts. The information that is held by the expert is extracted and a probabilistic representation of this knowledge is constructed. A promising avenue in psychological research is to incorporated experts’ prior knowledge in the statistical analysis. Systematic reviews on elicitation literature however suggest that it might be inappropriate to directly obtain distributional representations from experts. The literature qualifies experts’ performance on estimating elements of a distribution as unsatisfactory, thus reliably specifying the essential elements of the parameters of interest in one elicitation step seems implausible. Providing feedback within the elicitation process can enhance the quality of the elicitation and interactive software can be used to facilitate the feedback. Therefore, we propose to decompose the elicitation procedure into smaller steps with adjustable outcomes. We represent the tacit knowledge of experts as a location parameter and their uncertainty concerning this knowledge by a scale and shape parameter. Using a feedback procedure, experts can accept the representation of their beliefs or adjust their input. We propose a Five-Step Method which consists of (1) Eliciting the location parameter using the trial roulette method. (2) Provide feedback on the location parameter and ask for confirmation or adjustment. (3) Elicit the scale and shape parameter. (4) Provide feedback on the scale and shape parameter and ask for confirmation or adjustment. (5) Use the elicited and calibrated probability distribution in a statistical analysis and update it with data or to compute a prior-data conflict within a Bayesian framework. User feasibility and internal validity for the Five-Step Method are investigated using three elicitation studies.

show abstract

Experience with Expert Judgment: theTUDelft Expert Judgment Data

Cited by 15 publications

References 22 publications

Application and Evaluation of an Expert Judgment Elicitation Procedure for Correlations

Application and Evaluation of an Expert Judgment Elicitation Procedure for Correlations

The use of expert elicitation in environmental health impact assessment: a seven step procedure

Proposal for a Five-Step Method to Elicit Expert Judgment

Contact Info

Product

Resources

About