Expert Elicitation: Using the Classical Model to Validate Experts’ Judgments

Colson, Abigail; Cooke, Roger M.

doi:10.1093/reep/rex022

Cited by 105 publications

(102 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…SEJ is a formal method that calibrates experts based on their ability to estimate accurately their own uncertainty regarding relevant questions in their field of expertise. It penalizes both ignorance and overconfidence and has been shown to perform well in a variety of contexts (Oppenheimer et al, ; Colson & Cooke, ). It yields a probabilistic estimate while avoiding some of the biases that arise in consensus‐based expert judgements.…”

Section: Projections Of Relative Sea Level Changementioning

confidence: 99%

Usable Science for Managing the Risks of Sea‐Level Rise

et al. 2019

View full text Add to dashboard Cite

Sea-level rise sits at the frontier of usable climate climate change research, because it involves natural and human systems with long lags, irreversible losses, and deep uncertainty. For example, many of the measures to adapt to sea-level rise involve infrastructure and land-use decisions, which can have multigenerational lifetimes and will further influence responses in both natural and human systems. Thus, sea-level science has increasingly grappled with the implications of (1) deep uncertainty in future climate system projections, particularly of human emissions and ice sheet dynamics; (2) the overlay of slow trends and high-frequency variability (e.g., tides and storms) that give rise to many of the most relevant impacts; (3) the effects of changing sea level on the physical exposure and vulnerability of ecological and socioeconomic systems; and (4) the challenges of engaging stakeholder communities with the scientific process in a way that genuinely increases the utility of the science for adaptation decision making. Much fundamental climate system research remains to be done, but many of the most critical issues sit at the intersection of natural sciences, social sciences, engineering, decision science, and political economy. Addressing these issues demands a better understanding of the coupled interactions of mean and extreme sea levels, coastal geomorphology, economics, and migration; decision-first approaches that identify and focus research upon those scientific uncertainties most relevant to concrete adaptation choices; and a political economy that allows usable science to become used science.Plain Language Summary The impacts of sea-level rise pose growing threats to coastal communities, economies, and ecosystems, and decisions made today-in areas like land-use policies, coastal development, and infrastructure investment-will affect exposure and vulnerability for generations to come. Thus, the usability of sea-level science is a pressing concern. Ensuring usability requires grappling with deep uncertainty in long-term sea-level projections, the relationship between long-term trends and the impacts of short-lived extreme events, and the ways in which the physical coast, as well as people and ecosystems along the coast, respond to increasingly frequent flooding. At the same time, it also requires more extensive and deliberate stakeholder engagement throughout the scientific process, as well as cognizance of the political economy of linking stakeholder-engaged science to action.

show abstract

Section: Projections Of Relative Sea Level Changementioning

confidence: 99%

Usable Science for Managing the Risks of Sea‐Level Rise

et al. 2019

View full text Add to dashboard Cite

show abstract

“…However, a promising alternative approach to 'validation' is to pose seed questions, whereby experts are asked questions where the truth is known [24,25]. This provides a quantitative measure of the accuracy of each expert's opinion, and could be used to weight each expert's response to the main questions of interest.…”

Section: Discussionmentioning

confidence: 99%

“…2, panel A, region b). The ranges associated with each category of symptoms were drawn from published literature: 'mild' (1-10), 'moderate' (11)(12)(13)(14)(15)(16)(17)(18)(19)(20), 'moderate to severe' (21)(22)(23)(24)(25)(26)(27)(28)(29)(30)(31)(32)(33)(34)(35) and 'severe' (36-51). Figure 2, panel A, region a, shows illustrative results from an elicitation.…”

Section: Symptom Severity Scalementioning

confidence: 99%

A framework for extending trial design to facilitate missing data sensitivity analyses

Mason

Grieve

Richards-Belle

et al. 2020

BMC Med Res Methodol

View full text Add to dashboard Cite

Background: Missing data are an inevitable challenge in Randomised Controlled Trials (RCTs), particularly those with Patient Reported Outcome Measures. Methodological guidance suggests that to avoid incorrect conclusions, studies should undertake sensitivity analyses which recognise that data may be 'missing not at random' (MNAR). A recommended approach is to elicit expert opinion about the likely outcome differences for those with missing versus observed data. However, few published trials plan and undertake these elicitation exercises, and so lack the external information required for these sensitivity analyses. The aim of this paper is to provide a framework that anticipates and allows for MNAR data in the design and analysis of clinical trials. Methods: We developed a framework for performing and using expert elicitation to frame sensitivity analysis in RCTs with missing outcome data. The framework includes the following steps: first defining the scope of the elicitation exercise, second developing the elicitation tool, third eliciting expert opinion about the missing outcomes, fourth evaluating the elicitation results, and fifth analysing the trial data. We provide guidance on key practical challenges that arise when adopting this approach in trials: the criteria for identifying relevant experts, the outcome scale for presenting data to experts, the appropriate representation of expert opinion, and the evaluation of the elicitation results. The framework was developed within the POPPI trial, which investigated whether a preventive, complex psychological intervention, commenced early in ICU, would reduce the development of patient-reported post-traumatic stress disorder symptom severity, and improve health-related quality of life. We illustrate the key aspects of the proposed framework using the POPPI trial. Results: For the POPPI trial, 113 experts were identified with potentially suitable knowledge and asked to participate in the elicitation exercise. The 113 experts provided 59 usable elicitation questionnaires. The sensitivity analysis found that the results from the primary analysis were robust to alternative MNAR mechanisms. Conclusions: Future studies can adopt this framework to embed expert elicitation within the design of clinical trials. This will provide the information required for MNAR sensitivity analyses that examine the robustness of the trial conclusions to alternative, but realistic assumptions about the missing data.

show abstract

“…Essentially, this paper will follow the method that consists of two steps which were expert identification and evaluation procedure [30]. The process of expert identification and evaluation procedure are shown in Figure 1.…”

Section: Methodsmentioning

confidence: 99%

“…All the proposed components are validated through the expert review. After that, [30] suggested that mean approach can be used as one of a method to measure expert review. Therefore, a Likert scale analysis designed by [31] is used and the mean interpretation for the matched component also being modified to make it compatible with this research work.…”

Section: Evaluation Proceduresmentioning

confidence: 99%

The use of expert review in component development for customer satisfaction towards e-hailing

Idros

Mohamed

Jenal

2020

IJEECS

View full text Add to dashboard Cite

The emergence of e-hailing apps has increased the demand of customers because online transportation service is very convenient in this modern society. As e-hailing provides an online platform for booking a ride, this apps also functions to connect the customer and driver, while at the same time allowing the customers to interact with customer service that company has provided to keep track of the satisfaction. Competition among e-hailing companies has increased since customers nowadays are very influential and knowledgeable in terms of choosing any transportation services that they want. The purpose of this study is to identify customer satisfaction towards e-hailing by embedding value co-creation model. Value co-creation is one of the concepts in Service Science that has been proved to increase customer satisfaction. Value co-creation is focused on company-customer interaction that helps in maintaining a good customer relationship and improve the company’s performance. Expert identification and evaluation procedure are used as a technique to adapt the definition of value co-creation model towards e-hailing. Then, the other factor from previous studies will be proposed as a component to be listed under value co-creation model and reviewed by an expert.

show abstract

Expert Elicitation: Using the Classical Model to Validate Experts’ Judgments

Cited by 105 publications

References 40 publications

Usable Science for Managing the Risks of Sea‐Level Rise

Usable Science for Managing the Risks of Sea‐Level Rise

A framework for extending trial design to facilitate missing data sensitivity analyses

The use of expert review in component development for customer satisfaction towards e-hailing

Contact Info

Product

Resources

About