On the Comparison of Interval Forecasts

Askanazi, Ross; Diebold, Francis X.; Schorfheide, Frank; Shin, Minchul

doi:10.2139/ssrn.3225885

Cited by 5 publications

(3 citation statements)

References 23 publications

(24 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, they start with a certain scoring function of appeal to them and do not thoroughly characterise the functional which is elicited by this scoring function. We refer to Askanazi et al (2018) for an overview of interval forecasts, in which, however, mostly impossibility results are presented.…”

Section: Prediction Intervalsmentioning

confidence: 99%

Forecast evaluation of quantiles, prediction intervals, and other set-valued functionals

Fissler

Frongillo

Hlavinová

et al. 2021

Electron. J. Statist.

View full text Add to dashboard Cite

We introduce a theoretical framework of elicitability and identifiability of set-valued functionals, such as quantiles, prediction intervals, and systemic risk measures. A functional is elicitable if it is the unique minimiser of an expected scoring function, and identifiable if it is the unique zero of an expected identification function; both notions are essential for forecast ranking and validation, and M -and Z-estimation. Our framework distinguishes between exhaustive forecasts, being set-valued and aiming at correctly specifying the entire functional, and selective forecasts, content with solely specifying a single point in the correct functional. We establish a mutual exclusivity result: A set-valued functional can be either selectively elicitable or exhaustively elicitable or not elicitable at all. Notably, since quantiles are well known to be selectively elicitable, they fail to be exhaustively elicitable. We further show that the classes of prediction intervals and Vorob'ev quantiles turn out to be exhaustively elicitable, hence not selectively elicitable, but still selectively identifiable. In particular, we provide a mixture representation of elementary exhaustive scores, leading the way to Murphy diagrams. We establish that the shortest prediction interval and those specified by an endpoint or midpoint in general fail to be elicitable with respect to either notion, unless an endpoint is given via a quantile. We end with a comprehensive literature review on common practice in forecast evaluation of set-valued functionals.

show abstract

Section: Prediction Intervalsmentioning

confidence: 99%

Forecast evaluation of quantiles, prediction intervals, and other set-valued functionals

Fissler

Frongillo

Hlavinová

et al. 2021

Electron. J. Statist.

View full text Add to dashboard Cite

show abstract

“…Note that we assume that the two quantiles are known. In case we want to evaluate interval forecasts when the nominal coverage level is specified, but the quantiles on which intervals are based are not specified, one cannot employ the approach outlined here(Askanazi et al 2018).…”

mentioning

confidence: 99%

Evaluating probabilistic population forecasts

Keilman

2020

Preprint

View full text Add to dashboard Cite

We demonstrate how a probabilistic population forecast can be evaluated, when observations for the predicted variables become available. Statisticians have developed various scoring rules for that purpose, but there are hardly any applications in population forecasting literature. A scoring rule measures the distance between the probability distribution of the predicted variable, and the actual outcome. We use scoring rules that reward accuracy (the outcome is close to the expected value of the prediction) and sharpness (the predictive distribution has low variance, which makes it difficult to hit the target).We evaluate probabilistic population forecasts for France, the Netherlands, and Norway. For all three countries, we use results from the UPE-project ("Uncertain Population of Europe"). We inspect prediction intervals for population size in the period 2004-2019 and 3000 sample paths for population pyramids for the year 2010. For the Netherlands and for Norway, we compare the UPE-results with findings from the official probabilistic population forecast by Statistics Netherlands (2001-2019) and from a probabilistic forecast for Norway (1997-2019). All forecasts were computed using the cohort-component method and stochastically varying parameters for fertility, mortality and migration. We show that the UPE-forecasts for the Netherlands and for Norway performed better than the other forecasts for these two countries. The error in the jump-off population caused a bad score for the French forecast.We evaluate the 3000 UPE-simulations of the age and sex composition predicted for the year 2010. When normalized for population numbers in each age-sex category, the predictions for the Netherlands received the best scores, except for the oldest old. The age pattern for the Norwegian score reflects the under-prediction of immigration after the enlargement of the European Union in 2005.

show abstract

mentioning

confidence: 99%

Evaluating probabilistic population forecasts

Keilman

2020

Preprint

View full text Add to dashboard Cite

Statisticians have developed scoring rules for evaluating probabilistic forecasts against observations. However, there are very few applications in the literature on population forecasting. A scoring rule measures the distance between the predictive distribution and its outcome. We review scoring rules that reward accuracy (the outcome is close to the expectation of the distribution) and sharpness (the distribution has low variance, which makes it difficult to hit the target). We evaluate probabilistic population forecasts for France, the Netherlands, and Norway. Forecasts for total population size for the Netherlands and for Norway performed quite well. The error in the jump-off population caused a bad score for the French forecast. We evaluate the age and sex composition predicted for the year 2010. The predictions for the Netherlands received the best scores, except for the oldest old. The age pattern for the Norwegian score reflects the under-prediction of immigration after the enlargement of the European Union in 2005. JEL codes: C15, C44, J11,

show abstract

On the Comparison of Interval Forecasts

Cited by 5 publications

References 23 publications

Forecast evaluation of quantiles, prediction intervals, and other set-valued functionals

Forecast evaluation of quantiles, prediction intervals, and other set-valued functionals

Evaluating probabilistic population forecasts

Evaluating probabilistic population forecasts

Contact Info

Product

Resources

About