Using Expert Elicitation to Build Long-Term Projection Assumptions

Dion, Patrice; Galbraith, Nora; Sirag, Elham

doi:10.1007/978-3-030-42472-5_3

Cited by 8 publications

(8 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other limitations of this study include convenience sampling of experts and a small number of correlated questions. Participants were also asked to produce point estimates before they were asked to produce ranges, which is known to anchor responses toward the point estimate [49], so participants may have seemed more overconfident than they would have with other elicitation methods. Nevertheless, given the stark differences between expert and nonexpert accuracy and calibration levels, it seems unlikely that alternate elicitation methods would erase these differences.…”

Section: Discussionmentioning

confidence: 99%

How well did experts and laypeople forecast the size of the COVID-19 pandemic?

2021

View full text Add to dashboard Cite

Throughout the COVID-19 pandemic, social and traditional media have disseminated predictions from experts and nonexperts about its expected magnitude. How accurate were the predictions of ‘experts’—individuals holding occupations or roles in subject-relevant fields, such as epidemiologists and statisticians—compared with those of the public? We conducted a survey in April 2020 of 140 UK experts and 2,086 UK laypersons; all were asked to make four quantitative predictions about the impact of COVID-19 by 31 Dec 2020. In addition to soliciting point estimates, we asked participants for lower and higher bounds of a range that they felt had a 75% chance of containing the true answer. Experts exhibited greater accuracy and calibration than laypersons, even when restricting the comparison to a subset of laypersons who scored in the top quartile on a numeracy test. Even so, experts substantially underestimated the ultimate extent of the pandemic, and the mean number of predictions for which the expert intervals contained the actual outcome was only 1.8 (out of 4), suggesting that experts should consider broadening the range of scenarios they consider plausible. Predictions of the public were even more inaccurate and poorly calibrated, suggesting that an important role remains for expert predictions as long as experts acknowledge their uncertainty.

show abstract

Section: Discussionmentioning

confidence: 99%

How well did experts and laypeople forecast the size of the COVID-19 pandemic?

2021

View full text Add to dashboard Cite

show abstract

“…Statistics Canada (Dion, Galbraith, and Sirag 2020) used five-term metalogs to model expert opinion elicitations for future Canadian fertility rates. With an interactive spreadsheet, 17 experts were asked to adjust their quantile parameters with sliding bars until the resulting probability density graph accurately reflected their probabilistic beliefs about Canada fertility rate in 2043.…”

Section: Eliciting Expert Opinionmentioning

confidence: 99%

“…2.1.B) with using linear opinion pooling (Genest and Zidek 1986) to develop the combined-opinion PDF and CDF (Fig 2 .1 orange curves). A shortcoming of this method, in according to Dion, Galbraith, and Sirag (2020), is "Despite the fact that experts' responses are parametrized by metalog distributions, the resulting mixture distributions for fertility, mortality, and immigration are not metalog distributions, and do not belong to any defined parametric family. Characteristics such as central moments and quantiles are derived using numerical methods.…”

Section: Combining Expert Opinionmentioning

confidence: 99%

The Metalog Distributions: Virtually Unlimited Shape Flexibility, Combining Expert Opinion in Closed Form, and Bayesian Updating in Closed Form

Keelin¹,

Howard²

2021

Preprint

View full text Add to dashboard Cite

Users of probability distributions frequently need to convert data (empirical, simulated, or elicited) into a continuous probability distribution and to update that distribution when new data becomes available. Often, it is unclear which traditional probability distribution(s) to use, fitting to data is laborious and unsatisfactory, little insight emerges, and updating with Bayes rule is impractical. Here we offer an alternative -- a family of continuous probability distributions, fitting methods, and tools that: provide sufficient shape and boundedness flexibility to closely match virtually any probability distribution and most data sets; involve a single set of simple closed-form equations; stimulate potentially valuable insights when applied to empirical data; are simply fit to data with ordinary least squares; are easy to combine (as when weighting the opinion of multiple experts), and, under certain conditions, are easily updated in closed form according to Bayes rule when new data becomes available. The Bayesian updating method is presented in a way that is readily understandable as a fisherman updates his catch probabilities when changing the river on which he fishes. While metalog applications have been shown to improve decision-making, the methods and results herein are broadly applicable to virtually any use of continuous probability in any field of human endeavor. Diverse data sets may be explored and modeled in these new ways with freely available spreadsheets and tools.

show abstract

“…Another approach is to engage expert panels who are actively involved in the assumption formulation process (e.g. Dion et al 2020;Lutz 2009). The expert panel considers and discusses a wide range of factors which might increase or decrease migration (and other demographic variables) and then suggests future migration levels or distributions.…”

Section: International Migrationmentioning

confidence: 99%

A brief guide to producing a national population projection

Wilson

Rees

2021

Aust Popul Stud

View full text Add to dashboard Cite

Background There are surprisingly few resources available which offer an introductory guide to preparing a national population projection using a cohort-component model. Many demography textbooks cover projections quite briefly, and many academic papers on projections focus on advanced technical issues. Aims The aim of this paper is to provide a short and accessible guide to producing a national-scale population projection using the cohort-component model. Data and methods The paper describes the cohort-component model from a population accounting perspective, presents all the necessary projection calculations, and covers the key steps which form part of the projections preparation process – from gathering input data to validating outputs. An accompanying Excel workbook implements the model and contains example projections for Australia. Conclusions Calculating a national population projection using a cohort-component model involves fairly simple algebra, but the broader projections preparation process is more complex, and requires careful consideration and judgement.

show abstract

Using Expert Elicitation to Build Long-Term Projection Assumptions

Cited by 8 publications

References 34 publications

How well did experts and laypeople forecast the size of the COVID-19 pandemic?

How well did experts and laypeople forecast the size of the COVID-19 pandemic?

The Metalog Distributions: Virtually Unlimited Shape Flexibility, Combining Expert Opinion in Closed Form, and Bayesian Updating in Closed Form

A brief guide to producing a national population projection

Contact Info

Product

Resources

About