Assessing the reliability of ensemble forecasting systems under serial dependence

Abstract

The problem of testing the reliability of ensemble forecasting systems is revisited. A popular tool to assess the reliability of ensemble forecasting systems (for scalar verifications) is the rank histogram; this histogram is expected to be more or less flat, since, for a reliable ensemble, the ranks are uniformly distributed among their possible outcomes. Quantitative tests for flatness (e.g. Pearson's goodness‐of‐fit test) have been suggested; without exception, however, these tests assume the ranks to be a sequence of independent random variables, which is not the case in general, as can be demonstrated with simple toy examples. In this article, tests are developed that take the temporal correlations between the ranks into account. A refined analysis exploiting the reliability property shows that the ranks still exhibit strong decay of correlations. This property is key to the analysis, and the proposed tests are valid for general ensemble forecasting systems with minimal extraneous assumptions.

Stratified rank histograms for ensemble forecast verification under serial dependence

“…This can be harnessed to at least constrain the correlation structure of the ranks to some extent. The result of the presented analysis is a generalized χ 2 ‐test for the (joint) flatness of stratified rank histograms and thus for the reliability of ensemble forecasts, extending the results in Bröcker ().…”

“…We start with fixing some general notation. The general set‐up will be very similar to the one in Bröcker (). The verifications are modelled as a sequence { Y ( n ), n =1,…, N } of random variables with values in the real numbers, with the index n representing the time.…”

“…Irrespective of whether stratified or unstratified histograms are used, a rigorous testing methodology needs to take into account that a rank histogram will never be precisely flat even for a reliable forecasting system, and the random deviations from flatness need to be analyzed quantitatively. As has been noted by several authors (Pinson et al ., ; Wilks, ; Siegert et al ., ; Bröcker, ), a problem with forecast assessment in general is that the verification–forecast pairs (or in our case the ranks) are generally not independent, which renders this analysis very difficult. In particular, classical goodness‐of‐fit tests are not applicable to the flatness of rank histograms (stratified or unstratified) since the ranks are serially dependent.…”

“…The purpose of the present paper is to extend the approach taken in Bröcker (), which addresses this problem in the context of unstratified rank histograms, and extend it to stratified rank histograms. The approach rests on the observation that a reliable ensemble X ( n )={ X 1 ( n ),…, X K ( n )} provides (an approximation to) the distribution of Y ( n ), given what information was available to the forecaster at the initialisation time of the forecast X ( n ) (usually at time n − T , where T is the lead time).…”

Sequential Aggregation of Probabilistic Forecasts—Application to Wind Speed Ensemble Forecasts

“…In order to take into account this multiple testing, the false discovery rate is controlled by the Benjamini–Hochberg procedure (Benjamini & Hochberg, 1995; Benjamini & Yekutieli, 2001) with a control parameter of 0.01, denoted by in Benjamini and Hochberg (1995). Bröcker (2018) showed that when the observation ranks are serially dependent, the JP tests should be adapted to take into account this temporal dependency. In this study, the lag‐1 autocorrelation of the rank has a median of 0.2 over all the studied locations and lead times and is lower than 0.4 for most of the forecasting systems (raw, post‐processed and aggregated alike).…”