Verifying Forecasts Spatially

Abstract

An international project tests new spatial forecast verification methods to find out how they handle different types of forecast error and what they tell us about forecast performance.V erification of a forecast field presents many challenges, especially at higher resolutions. When assessing forecast performance at a single point, straightforward summary statistics [e.g., root-mean-square error (RMSE)] are meaningful because they give an intuitive notion of how well the forecasts matched the observations at that point. It is also straightforward to identify hits, misses, false alarms, and correct negatives, all of which give rise to numerous useful summary statistics [e.g., the probability of detection (POD) and the Gilbert skill score (GSS)] and diagnostics (e.g., the relative operating characteristic) of forecast performance (for more on traditional verification, see, e.g., Wilks 1995;Jolliffe and Stephenson 2003). However, when interest is in spatially coherent structures, these notions are not as simple to determine. Further, several new types of errors become relevant, which bring about a new set of verification questions. Are there spatial displacement errors? Does the scale-dependent variance of the forecast field match the spatial structure that was observed? Did the forecast under-or overpredict the spatial extent of a storm system? Are there orientation errors for specific structures in the field? At what scales do the forecasts have skill?Numerous methods have been proposed in recent years to address these issues, and some are already used regularly at meteorological centers around the world. Because the methods are relatively new, however, it is not always clear which approaches are best suited for answering particular questions about forecast performance. Which ones should be used for specific purposes, and which provide analogous information? How can uncertainty information be determined about the results? Do methods have any unintuitive characteristics about which a user should know? Such questions provided the impetus for the Spatial Forecast Verification Methods InterComparison project (ICP; details at www.ral.ucar. edu/projects/icp). Although not every approach is currently under study in the ICP, a reasonably representative subset is included, in most cases by the researchers who originally proposed them.test CAses. Several test cases have been provided in order to make head-to-head comparisons of how each method addresses forecast performance. So far, three sets of cases focused on quantitative precipitation forecasts (QPFs) are being used, with plans to incorporate more varied cases in the future. The cases VERIfyING fORECAStS SpAtIALLyBy eric Gilleland, david a. ahijevych, BarBara G. Brown, and elizaBeth e. eBert 1365 october 2010 AMerIcAN MeteoroLoGIcAL SocIetY | under study now include 24-h forecasts of 1-h precipitation from three configurations of the Weather Research and Forecasting (WRF) model, known perturbations of one of these cases, and simple geometric cases with prescribed spati...

On the forecast skill of a convection-permitting ensemble

“…In addition to the traditional statistical scores, precipitation forecasts are verified by spatial verification methods, which not only consider the exact match of forecast and verification values at individual points but also take into account the matching of forecasts and observations in terms of objects or spatial scales (Casati et al, 2008;Ahijevych et al, 2009;Gilleland et al, 2010). This is necessary as precipitation fields exhibit high spatial variability and discontinuity.…”

Spatial evaluation of volcanic ash forecasts using satellite observations

“…There are many neighbourhood skill scores described in the literature (see Ebert, 2008, andGilleland et al, 2010, for an overview). The method used in this paper is based on the FSS developed by Roberts and Lean (2008) to test the skill of high-resolution precipitation forecasts (e.g.…”

“…For example, if a volcanic plume is forecast to have the perfect shape but is displaced due to small errors in wind speed, metrics that compare each point in space and time (known as point-by-point in this paper) would yield low values as the feature is not in the correct place at the correct time. This problem has given rise to a host of other techniques to evaluate model skill, each suitable for evaluating different aspects of the forecast (see Gilleland et al, 2010, for a review of these techniques). In this paper the spatial accuracy of the VATD forecasts is being assessed and therefore a neighbourhood technique is used.…”

Comparison of selected methods of analysis for reconstructed fields of precipitation in climate scenarios over Poland

“…This problem also occurs in the verification of numerical weather forecasts and has been studied in the project BSpatial Forecast Verification Methods Inter-Comparison Project^ (Ahijevych et al 2009a, b). In the view of many publications related to this project, there are more appropriate methods to analyze the compatibility of precipitation fields (Gilleland et al 2010;Ahijevych et al 2009a, b;Gilleland et al 2009). The methods and ideas adopted from this project can be applied to the evaluation of climate models.…”