Towards improving the framework for probabilistic forecast evaluation

Smith, Leonard A.; Suckling, Emma; Thompson, Erica; Maynard, Trevor; Du, Hailiang

doi:10.1007/s10584-015-1430-2

Cited by 26 publications

(31 citation statements)

References 32 publications

(48 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It assigns value not only to the predicted probability of an observation but also to the distance of a predicted probability mass from an observation. It is therefore relatively robust to specific functional forms of the density forecasts (30) and allows for comparison with point and ensemble forecasts (31,32) …”

Section: Resultsmentioning

confidence: 99%

“…Most of these methods take an integrated approach to forecast the whole distribution, including the best estimate. The empirical methods we use here instead allow analysts or forecast users to attach an uncertainty distribution to a preexisting point forecast.The importance of density forecast evaluation has been discussed by several authors (17,(28)(29)(30). When methods are chosen to generate probabilistic energy forecasts, such evaluation is often omitted.…”

mentioning

confidence: 99%

“…The importance of density forecast evaluation has been discussed by several authors (17,(28)(29)(30). When methods are chosen to generate probabilistic energy forecasts, such evaluation is often omitted.…”

mentioning

confidence: 99%

“…As it is a measure of both calibration and sharpness, we use the continuous ranked probability score (CRPS) (30)(31)(32) to compare density forecasts. For point forecast evaluation we work with the average prediction error, here the mean absolute percentage error (MAPE), and the transformed mean absolute logarithmic error (MALE) for prices (Materials and Methods).…”

mentioning

confidence: 99%

See 3 more Smart Citations

Empirical prediction intervals improve energy forecasting

Kaack

Apt

Morgan

et al. 2017

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

Hundreds of organizations and analysts use energy projections, such as those contained in the US Energy Information Administration (EIA)'s Annual Energy Outlook (AEO), for investment and policy decisions. Retrospective analyses of past AEO projections have shown that observed values can differ from the projection by several hundred percent, and thus a thorough treatment of uncertainty is essential. We evaluate the out-of-sample forecasting performance of several empirical density forecasting methods, using the continuous ranked probability score (CRPS). The analysis confirms that a Gaussian density, estimated on past forecasting errors, gives comparatively accurate uncertainty estimates over a variety of energy quantities in the AEO, in particular outperforming scenario projections provided in the AEO. We report probabilistic uncertainties for 18 core quantities of the AEO 2016 projections. Our work frames how to produce, evaluate, and rank probabilistic forecasts in this setting. We propose a log transformation of forecast errors for price projections and a modified nonparametric empirical density forecasting method. Our findings give guidance on how to evaluate and communicate uncertainty in future energy outlooks.forecast uncertainty | density forecasts | scenarios | continuous ranked probability score | fan chart P rojections of quantities such as electricity and fuel demands, commodity prices, and specific energy consumption and production rates are widely used to inform private and public investment decisions, long-term strategies, and policy analysis (1-3). Policy analysts and decision makers often use modeled projections as forecasts with little or no discussion about the associated uncertainty (2, 4, 5). [Energy outlooks are often referred to as projections because they refrain from incorporating future policy changes into the reference scenario. In contrast, the term forecast denotes a best estimate allowing for all changes of the state of the world (6). While we are aware of this difference, our analysis treats the reference scenario as the best estimate forecast. We use the terms forecast and projection interchangeably.] Here we are concerned with national-scale forecasts in the energy industry that span a range from years to decades. Two of the most influential sets of energy projections are those of the US Energy Information Administration (EIA) and the International Energy Agency (IEA), complemented by those made by private oil and gas companies, such as Shell, ExxonMobil, and Statoil. When assessed retrospectively, such energy projections have sometimes shown very large deviations from the realized values (7-9). Providing information on the likely uncertainty associated with such projections would help individuals and organizations use them in a more informed manner (10-12).All of the energy outlooks mentioned above provide point projections without a probabilistic treatment of uncertainty. Often, point forecasts are labeled as a "reference scenario" and are accompanied by alternative scenarios. ...

show abstract

Section: Resultsmentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

Empirical prediction intervals improve energy forecasting

Kaack

Apt

Morgan

et al. 2017

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

show abstract

“…In this work, we have also argued from a perspective on forecast quality assessment that emphasizes the importance of decision support. Forecast skill scores can often be difficult to interpret in terms of decision support by someone wishing to ascertain the superiority of some forecast set over another; and since different choices of skill score do not agree on a unique sort order of forecast excellence, the process of preferring some forecast sets over others on the basis of skill has something of a beauty-contest air about it [37]. We have seen that one can rate the performance of forecast sets concretely, in terms of their ability to consistently win bets against other forecast sets.…”

Section: Discussionmentioning

confidence: 99%

Probabilistic recalibration of forecasts

Graziani

Rosner

Adams³

et al. 2021

International Journal of Forecasting

View full text Add to dashboard Cite

We present a scheme by which a probabilistic forecasting system whose predictions have poor probabilistic calibration may be recalibrated by incorporating past performance information to produce a new forecasting system that is demonstrably superior to the original, in that one may use it to consistently win wagers against someone using the original system. The scheme utilizes Gaussian process (GP) modeling to estimate a probability distribution over the Probability Integral Transform (PIT) of a scalar predictand. The GP density estimate gives closed-form access to information entropy measures associated with the estimated distribution, which allows prediction of winnings in wagers against the base forecasting system. A separate consequence of the procedure is that the recalibrated forecast has a uniform expected PIT distribution. A distinguishing feature of the procedure is that it is appropriate even if the PIT values are not i.i.d. The recalibration scheme is formulated in a framework that exploits the deep connections between information theory, forecasting, and betting. We demonstrate the effectiveness of the scheme in two case studies: a laboratory experiment with a nonlinear circuit and seasonal forecasts of the intensity of the El Niño-Southern Oscillation phenomenon.

show abstract

Data‐Driven Model Uncertainty Estimation in Hydrologic Data Assimilation

Pathiraja

Moradkhani

Marshall

et al. 2018

Water Resources Research

View full text Add to dashboard Cite

The increasing availability of earth observations necessitates mathematical methods to optimally combine such data with hydrologic models. Several algorithms exist for such purposes, under the umbrella of data assimilation (DA). However, DA methods are often applied in a suboptimal fashion for complex real‐world problems, due largely to several practical implementation issues. One such issue is error characterization, which is known to be critical for a successful assimilation. Mischaracterized errors lead to suboptimal forecasts, and in the worst case, to degraded estimates even compared to the no assimilation case. Model uncertainty characterization has received little attention relative to other aspects of DA science. Traditional methods rely on subjective, ad hoc tuning factors or parametric distribution assumptions that may not always be applicable. We propose a novel data‐driven approach (named SDMU) to model uncertainty characterization for DA studies where (1) the system states are partially observed and (2) minimal prior knowledge of the model error processes is available, except that the errors display state dependence. It includes an approach for estimating the uncertainty in hidden model states, with the end goal of improving predictions of observed variables. The SDMU is therefore suited to DA studies where the observed variables are of primary interest. Its efficacy is demonstrated through a synthetic case study with low‐dimensional chaotic dynamics and a real hydrologic experiment for one‐day‐ahead streamflow forecasting. In both experiments, the proposed method leads to substantial improvements in the hidden states and observed system outputs over a standard method involving perturbation with Gaussian noise.

show abstract

Towards improving the framework for probabilistic forecast evaluation

Cited by 26 publications

References 32 publications

Empirical prediction intervals improve energy forecasting

Empirical prediction intervals improve energy forecasting

Probabilistic recalibration of forecasts

Data‐Driven Model Uncertainty Estimation in Hydrologic Data Assimilation

Contact Info

Product

Resources

About