A new equitable score suitable for verifying precipitation in numerical weather prediction

Abstract

A new equitable score is developed for monitoring precipitation forecasts and for guiding forecast system development. To accommodate the difficult distribution of precipitation, the score measures the error in 'probability space' through use of the climatological cumulative distribution function. For sufficiently skilful forecasting systems, the new score is less sensitive to sampling uncertainty than other established scores. It is therefore called here the 'Stable Equitable Error in Probability Space' (SEEPS). Weather is partitioned into three categories: 'dry', 'light precipitation' and 'heavy precipitation'. SEEPS adapts to the climate of the region in question so that it assesses the salient aspects of the local weather, encouraging 'refinement' and discouraging 'hedging'. To permit continuous monitoring of a system with resolution increasing in time, forecasts are verified against point observations. With some careful choices, observation error and lack of representativeness of model grid-box averages are found to have relatively little impact. SEEPS can identify key forecasting errors including the overprediction of drizzle, failure to predict heavy large-scale precipitation and incorrectly locating convective cells. Area averages are calculated taking into account the observation density. A gain of ∼2 days, at lead times of 3-9 days, over the last 14 years is found in extratropical scores of forecasts made at the European Centre for Medium-Range Weather Forecasts (ECMWF). This gain is due to system improvements, not the increased amount of data assimilated. SEEPS may also be applicable for verifying other quantities that suffer from difficult spatio-temporal distributions. Copyright c 2010 Royal Meteorological Society Key Words: equitability; probability space; sampling uncertainty; refinement; hedging

How do I know if my forecasts are better? Using benchmarks in hydrological ensemble prediction

“…Thus the importance of using benchmarks that are known and understood is essential in assessing how 'good' forecasts are (Seibert, 2001;Garrick et al, 1978;Martinec and Rango, 1989;Murphy and Winkler, 1987;Schaefli and Gupta, 2007). There is a wealth of literature on comparing models or forecasts, developing techniques to evaluate skill and on the use of benchmarks in hydro-meteorological forecasting Dawson et al, 2007;Ewen, 2011;Gordon et al, 2000;Nicolle et al, 2013;Pappenberger and Beven, 2004;Pappenberger et al, 2011a;Rodwell et al, 2010;Rykiel, 1996). Although there is surprisingly little consensus on which benchmarks are most suited for which application, benchmark suitability has been found to depend on the model structure used in the forecasting system, the season, catchment characteristics, river regime and flow conditions.…”

The Benefits of Global High Resolution for Climate Simulation: Process Understanding and the Enabling of Stakeholder Decisions at the Regional Scale

“…An example of monitoring progress in NWP, citing resolution as one aspect of improvements in skills scores, is shown in Fig. 10 of Rodwell et al (2010). A more general, high-level review of the benefits of resolution in NWP models is provided by Wedi (2014).…”

Data assimilation in atmospheric chemistry models: current status and future prospects for coupled chemistry meteorology models

“…See for example Table 6 in Zhang et al (2012a) for a non-exhaustive list of such parameters. Ongoing research continuously adds to such procedures (e.g., Rodwell et al, 2010;Ferro and Stevenson, 2011). Similar procedures may be used with CCMM to evaluate the improvement provided by data assimilation in a forecasting mode (e.g., see case studies in Sects.…”