Forecast verification: Its Complexity and Dimensionality

ABSTRACT:Although there are numerous reasons for performing a verification analysis, there are usually two general questions that are of interest: are the forecasts good, and can we be confident that the estimate of forecast quality is not misleading? When calculating a verification score, it is not usually obvious how the score can answer either of these questions. Some procedures for attempting to answer the questions are reviewed, with particular focus on p-values and confidence intervals. P -values are shown to be rather unhelpful in answering either question, especially when applied to probabilistic verification scores, and confidence intervals are to be preferred. However, confidence intervals cannot reveal biases in the value of a score that arises from an inadequate experimental design for testing on truly out-of-sample observations. Some specific problems with cross validation are highlighted. Finally, in the interests of increasing the insight into forecast strengths and weaknesses and in pointing towards methods for improving forecast quality, a plea is made for a more discriminating selection of verification procedures than has been adopted to date.

Section: Understandabilitymentioning

confidence: 99%

Understanding forecast verification statistics

Mason

2008

“…For both the cool and warm seasons, the MOS forecasts are judged to be better according to the SS statistic, while the SUB forecasts are superior in both cases according to the A ROC . This conflicting result should not be especially surprising in view of the fact that compressing a high-dimensional verification problem into a single scalar value will necessarily obscure some important information (Murphy, 1991;Murphy & Winkler, 1987).…”

mentioning

confidence: 99%

A skill score based on economic value for probability forecasts

Wilks

2001

122

134

An approach to evaluating probability forecasts for dichotomous events, based on their economic value over all possible cost/loss ratio decision problems, is proposed. The resulting Value Score (VS) curve shows non‐dimensionalised relative economic value as a function of the cost/loss ratios for different decision‐makers, over their full meaningful range. The VS curve is similar in terms of computational mechanics and graphical display to the Relative Operating Characteristic (ROC) curve, but the ROC curve is shown to be insensitive to either conditional or unconditional biases and thus to reflect potential rather than actual skill. The possibility of collapsing the VS curve into a single scalar score is addressed, and it is shown that the results can depend very strongly on the assumed distribution of cost/loss ratios in the community of forecast users. Copyright © 2001 Royal Meteorological Society

“…This , 2002). The range of these four forecast measures for this event allows us to evaluate different aspects of the forecast quality, and thus summarize the performance of the forecasts more adequately (Murphy, 1991). The average performance of the control forecast for the July and August 2002 Danube floods to observations can be seen in Table VII.…”

Section: Discussionmentioning

confidence: 99%

“…The outline of the concept was first suggested by Murphy (1991Murphy ( , 1996 and has been used previously in the context of hedging (Jolliffe, 2007). Here, an approach and application to a hydrometeorological problem is described.…”

Section: A Six-step Approach To Screen Performance Measuresmentioning

confidence: 99%

Evaluating forecasts of extreme events for hydrological applications: an approach for screening unfamiliar performance measures

Cloke

Pappenberger

2008

Many different performance measures have been developed to evaluate field predictions in meteorology. However, a researcher or practitioner encountering a new or unfamiliar measure may have difficulty in interpreting its results, which may lead to them avoiding new measures and relying on those that are familiar. In the context of evaluating forecasts of extreme events for hydrological applications, this article aims to promote the use of a range of performance measures. Some of the types of performance measures that are introduced in order to demonstrate a six-step approach to tackle a new measure. Using the example of the European Centre for Medium-Range Weather Forecasts (ECMWF) ensemble precipitation predictions for the Danube floods of July and August 2002, to show how to use new performance measures with this approach and the way to choose between different performance measures based on their suitability for the task at hand is shown.