Fuzzy verification of high‐resolution gridded forecasts: a review and proposed framework

Abstract

High-resolution forecasts from numerical models can look quite realistic and provide the forecaster with very useful guidance. However, when verified using traditional metrics they often score quite poorly because of the difficulty of predicting an exact match to the observations at high resolution. 'Fuzzy' verification rewards closeness by relaxing the requirement for exact matches between forecasts and observations. The key to the fuzzy approach is the use of a spatial window or neighbourhood surrounding the forecast and/or observed points. The treatment of the data within the window may include averaging (upscaling), thresholding, or generation of a PDF, depending on the particular fuzzy method used and its implicit decision model concerning what makes a good forecast. The size of the neighbourhood can be varied to provide verification results at multiple scales, thus allowing the user to determine at which scales the forecast has useful skill.This article describes a framework for fuzzy verification that incorporates several fuzzy verification methods. It is demonstrated on a high-resolution precipitation forecast from the United Kingdom (UK) and the results interpreted to show the additional information that can be gleaned from this approach.

Deterministic and fuzzy verification methods for a hierarchy of numerical models

“…The forecast frequency of an event is computed in a neighbourhood. To compare this probabilistic information with the local occurrence of the observed event, a BSS with a persistence reference BSS SO is computed as in Ebert (2008). A second strategy is to compare this information to the observed frequency in the same neighbourhood by a second BSS score with a persistence reference BSS NO (Ebert, 2008).…”

“…One of the main problems in the verification of high resolution models (HR model) is the 'double penalty' (Theis et al, 2005;Ebert, 2008). If an HR model forecasts an observed small scale phenomenon more realistically than a low resolution model (LR model), but misplaces it slightly, it is penalized twice: once for missing the actual feature and again for forecasting it where it is not.…”

“…Probabilistic, fuzzy approaches (Theis et al, 2005;Roberts and Lean, 2008) and scale analysis verification approaches (Casati et al, 2004;Mittermaier, 2006) address the double penalty issue. Ebert (2008) lists and compares some of these new methods for a case of strong convection over the United Kingdom. A more general review by Casati 192 M. AMODEI AND J. STEIN stretched coordinate system which is equal to 23 km over France and 133 km over New Zealand during the period of this study.…”

A nudging scheme to assimilate satellite brightness temperature in a meteorological model: Impact on representation of African mesoscale convective systems

“…Fractional skill score (FSS; Roberts and Lean, 2008) was used to evaluate the performance of the model, in terms of CTBT (and rainfall in the following section). FSS scores belong to the verification methods based on fuzzy logic (Ebert, 2008), which relax the requirement for exact matches between forecasts and observations, using instead a spatial window or neighbourhood surrounding the forecast and observed points.…”

Evaluation concepts to compare observed and simulated deposition areas of mass movements

“…Nevertheless, physical models have been applied worldwide for hazard assessments for different kind of mass movements, whereas simulation results respectively evaluation concepts are often based on subjective or expert based rules. A systematic review of 75 peer reviewed articles 5 from 1995 to 2015 which inverse modelled real events of different types of mass flows revealed that there are no standard approaches for testing the accuracy of observed and simulated deposition patterns of mass movement simulation models, unlike in statistical, computational or meteorological modelling where a lot of different error measures exist [1,4,14].…”