Intercomparison of Spatial Forecast Verification Methods

Abstract

Advancements in weather forecast models and their enhanced resolution have led to substantially improved and more realistic-appearing forecasts for some variables. However, traditional verification scores often indicate poor performance because of the increased small-scale variability so that the true quality of the forecasts is not always characterized well. As a result, numerous new methods for verifying these forecasts have been proposed. These new methods can mostly be classified into two overall categories: filtering methods and displacement methods. The filtering methods can be further delineated into neighborhood and scale separation, and the displacement methods can be divided into features based and field deformation. Each method gives considerably more information than the traditional scores, but it is not clear which method(s) should be used for which purpose.A verification methods intercomparison project has been established in order to glean a better understanding of the proposed methods in terms of their various characteristics and to determine what verification questions each method addresses. The study is ongoing, and preliminary qualitative results for the different approaches applied to different situations are described here. In particular, the various methods and their basic characteristics, similarities, and differences are described. In addition, several questions are addressed regarding the application of the methods and the information that they provide. These questions include (i) how the method(s) inform performance at different scales; (ii) how the methods provide information on location errors; (iii) whether the methods provide information on intensity errors and distributions; (iv) whether the methods provide information on structure errors; (v) whether the approaches have the ability to provide information about hits, misses, and false alarms; (vi) whether the methods do anything that is counterintuitive; (vii) whether the methods have selectable parameters and how sensitive the results are to parameter selection; (viii) whether the results can be easily aggregated across multiple cases; (ix) whether the methods can identify timing errors; and (x) whether confidence intervals and hypothesis tests can be readily computed.

A critical assessment of surface cloud observations and their use for verifying cloud forecasts

“…The issues with verifying km-scale precipitation forecasts are well established in the literature, with many developed spatial verification methods rewarding close forecasts (e.g. Gilleland et al, 2009, provide a review), and it is hypothesised that the results shown here are similar to what is seen for high-resolution precipitation forecasts verified against gauges. A preliminary test of this hypothesis is offered here, using the 'minimum coverage' spatial method proposed by Damrath (2004) which enables a point observation to be compared to a forecast neighbourhood (instead of just the nearest grid point).…”

Predictive skill of a subset of models participating in D‐PHASE in the COPS region

“…In order to allow for an evaluation of the small-scale structure of precipitation, new verification techniques have been developed in recent years. According to Gilleland et al (2009), the different methods can be grouped into four categories: (i) neighbourhood (or fuzzy), (ii) scale separation (or scale decomposition), (iii) feature-based (or objectbased), and (iv) field deformation.…”

Distinguishing high and low flow domains in urban drainage systems 2 days ahead using numerical weather prediction ensembles

“…This concept has also been used for forecast verification purposes, e.g. (Ebert, 2008), (Gilleland et al, 2009) and (Ebert, 2009). …”

“…If the magnitude of a weather event is correctly forecasted but slightly displaced in space then the model will be penalized twice: once for missing the observations and once again for giving a false alarm; this is known as the ''double penalty'' (Michaelides, 2008). Hence the value of using a finer spatial resolution may be underestimated when using traditional verification procedures, and two overall categories of new performance evaluation methods have therefore been developed: filtering methods and displacement methods (Gilleland et al, 2009). …”