Enforcing calibration in ensemble postprocessing

Abstract

Desirable attributes of probability forecasts are maximal sharpness, consistent with calibration (‘reliability’). The usual procedure of optimizing ensemble‐postprocessing parameters by minimizing a proper scoring rule such as the continuous ranked probability score or the ‘ignorance’ (i.e. negative log‐likelihood) does not guarantee the necessary calibration condition, potentially compromising the value of the resulting forecasts to users. The calibration condition can be enforced by including a miscalibration penalty in the loss function to be minimized in parameter estimation. The procedure is illustrated using ensemble forecasts for minimum temperatures and wind speeds, postprocessed using member‐by‐member algorithms.

New approaches to postprocessing of multi‐model ensemble forecasts

“…() that sharpness should be improved only while respecting forecast calibration. Wilks () proposes the introduction of a penalty function in the optimization step to ensure that calibration, rather than sharpness, is maximized; however, this approach has not been widely adopted, so we have chosen to use the standard form of the algorithm provided by the ensembleMOS package in R (Yuen et al . ).…”

“…Fitting by CRPS minimization means that NGR forecasts generally perform well when assessed via single scoring rules – particularly, of course, the CRPS. However, recent research suggests that optimization by CRPS minimization can lead to forecasts that are sharper (having lower variance) than competitors, but at the cost of calibration (Wilks, ): this contravenes the maxim of Gneiting et al . () that sharpness should be improved only while respecting forecast calibration.…”

“…By integrating the Brier score over all possible thresholds, we obtain the analogue for continuous probabilistic forecasts: the Continuous Ranked Probability Score (CRPS; Hersbach ), a negatively oriented scoring rule with lower scores indicating better overall forecast performance. The CRPS is generally highly correlated with the MAE, but rewards sharpness and calibration of the predictive density as well as accuracy; in fact, it has been shown that it may reward sharper forecasts in preference to well‐calibrated ones (Wilks, ), so care should be taken when interpreting the CRPS. The CRPS can be extended to the multivariate case in the form of the energy score (Gneiting et al .…”

“…Fitting by CRPS minimization means that NGR forecasts generally perform well when assessed via single scoring rules -particularly, of course, the CRPS. However, recent research suggests that optimization by CRPS minimization can lead to forecasts that are sharper (having lower variance) than competitors, but at the cost of calibration (Wilks, 2018): this contravenes the maxim of Gneiting et al (2008) that sharpness should be improved only while respecting forecast calibration. Wilks (2018) proposes the introduction of a penalty function in the optimization step to ensure that calibration, rather than sharpness, is maximized; however, this approach has not been widely adopted, so we have chosen to use the standard form of the algorithm provided by the ensembleMOS package in R (Yuen et al, 2017).…”

A Generic Method for Density Forecasts Recalibration

“…Bröcker [1] generalized this result to any proper score, that is any score which is minimal if the forecasted probability distribution is the true one (w.r.t the available information). Recently, Wilks [14] proposed to add an extra miscalibration penalty, in order to enforce calibration in ensemble postprocessing. Nevertheless, even if the score we use mixes calibration and sharpness, the framework is essentially different from the first one.…”