Validation in the Classical Model

Cooke, Roger M.

doi:10.1007/978-3-319-65052-4_3

Cited by 6 publications

(3 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Decision makers often rely on expert judgement to fill knowledge gaps (Cooke 1991;Sutherland and Burgman 2015;Cooke et al 2018). Expert judgement has provided estimates and predictions of key unknowns in fields as diverse as nuclearpower safety (Cooke 1991), volcanic eruptions (Aspinall 2010), climate change (Bamber et al 2019), and biodiversity loss (Schlapfer et al 1999;Sala et al 2000).…”

mentioning

confidence: 99%

Expert perspectives on global biodiversity loss and its drivers and impacts on people

Isbell

Balvanera

Mori

et al. 2022

Frontiers in Ecol & Environ

View full text Add to dashboard Cite

Despite substantial progress in understanding global biodiversity loss, major taxonomic and geographic knowledge gaps remain. Decision makers often rely on expert judgement to fill knowledge gaps, but are rarely able to engage with sufficiently large and diverse groups of specialists. To improve understanding of the perspectives of thousands of biodiversity experts worldwide, we conducted a survey and asked experts to focus on the taxa and freshwater, terrestrial, or marine ecosystem with which they are most familiar. We found several points of overwhelming consensus (for instance, multiple drivers of biodiversity loss interact synergistically) and important demographic and geographic differences in specialists’ perspectives and estimates. Experts from groups that are underrepresented in biodiversity science, including women and those from the Global South, recommended different priorities for conservation solutions, with less emphasis on acquiring new protected areas, and provided higher estimates of biodiversity loss and its impacts. This may in part be because they disproportionately study the most highly threatened taxa and habitats. Front Ecol Environ 2022;

show abstract

mentioning

confidence: 99%

Expert perspectives on global biodiversity loss and its drivers and impacts on people

Isbell

Balvanera

Mori

et al. 2022

Frontiers in Ecol & Environ

View full text Add to dashboard Cite

show abstract

“…A further complication arises in understanding scoring rules because the terms used to describe judgment, such as “calibration,” “accuracy,” and “overconfidence,” are used interchangeably and may refer to different concepts (Lichtenstein and Fischhoff , Lichtendahl et al. , Cooke , Hemming et al. ).…”

Section: Methodsmentioning

confidence: 99%

Weighting and aggregating expert ecological judgments

Hemming

Hanea

Walshe

et al. 2020

Ecological Applications

View full text Add to dashboard Cite

Citation: Hemming, V., A. M. Hanea, T. Walshe, and M. A. Burgman. 2020. Weighting and aggregating expert ecological judgments. Ecological Applications 30(4):Abstract. Performance weighted aggregation of expert judgments, using calibration questions, has been advocated to improve pooled quantitative judgments for ecological questions. However, there is little discussion or practical advice in the ecological literature regarding the application, advantages or challenges of performance weighting. In this paper we (1) illustrate how the IDEA protocol with four-step question format can be extended to include performance weighted aggregation from the Classical Model, and (2) explore the extent to which this extension improves pooled judgments for a range of performance measures. Our case study demonstrates that performance weights can improve judgments derived from the IDEA protocol with four-step question format. However, there is no a-priori guarantee of improvement. We conclude that the merits of the method lie in demonstrating that the final aggregation of judgments provides the best representation of uncertainty (i.e., validation), whether that be via equally weighted or performance weighted aggregation. Whether the time and effort entailed in performance weights can be justified is a matter for decision-makers. Our case study outlines the rationale, challenges, and benefits of performance weighted aggregations. It will help to inform decisions about the deployment of performance weighting and avoid common pitfalls in its application.

show abstract

“…We outline the three most widely used methods of expert judgement aggregation: the Classical Method [38], SHELF [39], and the modified Delphi method [40]. These reflect mathematical, consensus, and hybrid approaches, respectively.…”

mentioning

confidence: 99%

In Search of Complementarity: Insights from an Exercise in Quantifying Qualitative Energy Futures

et al. 2022

View full text Add to dashboard Cite

In this study, we considered a bridging strategy between qualitative and quantitative research with the aim of achieving complementarity. A pilot case study using the Sheffield Elicitation Framework “SHELF” to estimate appropriate inputs for a quantitative energy systems model (based on a qualitative energy future scenario) was used to gain insights. Of novelty are the ethnographic insights of an example translation procedure as well as the methodological approach of the translation procedure itself. This paper reports the findings from this exercise concerning the practicalities of applying such a technique and the observations from the expert elicitation process itself. Based on this pilot, we make two recommendations. The first is the importance of devising a strategy in projects, and research programmes, where bridging between qualitative and quantitative research activities would be most effective. The second is that observations of discussions during the expert elicitation process provide value in the provenance of the estimates for quantitative modelling purposes and provide considerations for further development of qualitative future scenarios.

show abstract

Validation in the Classical Model

Cited by 6 publications

References 29 publications

Expert perspectives on global biodiversity loss and its drivers and impacts on people

Expert perspectives on global biodiversity loss and its drivers and impacts on people

Weighting and aggregating expert ecological judgments

In Search of Complementarity: Insights from an Exercise in Quantifying Qualitative Energy Futures

Contact Info

Product

Resources

About