A skill score based on economic value for probability forecasts

Wilks, Daniel S.

doi:10.1017/s1350482701002092

Cited by 122 publications

(141 citation statements)

References 33 publications

Supporting

Mentioning

139

Contrasting

Order By: Relevance

“…There has been some concern, however, that these measures constituted such a simplifying reduction in dimensionality and information associated with the multifaceted nature of forecast quality that, on their own, they were insufficient in describing the forecast quality of a forecasting system (Ehrendorfer and Murphy, 1988;Murphy and Ye, 1990;Murphy, 1993;Wilks, 1995;Wilks, 2001;Hartmann et al, 2002). One of the fundamental characteristics of these measures is that they derive a categorical or single-value forecast from the forecast distribution and contrast this value with the ultimately realized single observed value.…”

Section: Introductionmentioning

confidence: 99%

On measuring quality of a probabilistic commodity forecast for a system that incorporates seasonal climate forecasts

Potgieter¹,

Everingham

Hammer

2003

Intl Journal of Climatology

View full text Add to dashboard Cite

Regional commodity forecasts are being used increasingly in agricultural industries to enhance their risk management and decision-making processes. These commodity forecasts are probabilistic in nature and are often integrated with a seasonal climate forecast system. The climate forecast system is based on a subset of analogue years drawn from the full climatological distribution. In this study we sought to measure forecast quality for such an integrated system. We investigated the quality of a commodity (i.e. wheat and sugar) forecast based on a subset of analogue years in relation to a standard reference forecast based on the full climatological set. We derived three key dimensions of forecast quality for such probabilistic forecasts: reliability, distribution shift, and change in dispersion. A measure of reliability was required to ensure no bias in the forecast distribution. This was assessed via the slope of the reliability plot, which was derived from examination of probability levels of forecasts and associated frequencies of realizations. The other two dimensions related to changes in features of the forecast distribution relative to the reference distribution. The relationship of 13 published accuracy/skill measures to these dimensions of forecast quality was assessed using principal component analysis in case studies of commodity forecasting using seasonal climate forecasting for the wheat and sugar industries in Australia. There were two orthogonal dimensions of forecast quality: one associated with distribution shift relative to the reference distribution and the other associated with relative distribution dispersion. Although the conventional quality measures aligned with these dimensions, none measured both adequately. We conclude that a multi-dimensional approach to assessment of forecast quality is required and that simple measures of reliability, distribution shift, and change in dispersion provide a means for such assessment. The analysis presented was also relevant to measuring quality of probabilistic seasonal climate forecasting systems. The importance of retaining a focus on the probabilistic nature of the forecast and avoiding simplifying, but erroneous, distortions was discussed in relation to applying this new forecast quality assessment paradigm to seasonal climate forecasts.

show abstract

Section: Introductionmentioning

confidence: 99%

On measuring quality of a probabilistic commodity forecast for a system that incorporates seasonal climate forecasts

Potgieter¹,

Everingham

Hammer

2003

Intl Journal of Climatology

View full text Add to dashboard Cite

show abstract

“…Value scores are based on a simple cost/loss model (Wilks 2001). The value score (VS) of a forecast system can be interpreted as the expected economic value of the forecasts of interest as a fraction of the value of perfect forecasts relative to climatological forecasts, or as a percentage improvement in value between climatological and perfect information, as a function of the cost/loss ratio, for 0 < C/L < 1 (Wilks 2001 The value score was estimated using the following equations (Wilks 2001).…”

Section: Value Scorementioning

confidence: 99%

“…That is, for any given forecast probability except climatology, the outcome was closer to the climatological value. Overconfident forecasting can impart negative economic value to users of the forecasts (Wilks 2001). …”

Section: Reliabilitymentioning

confidence: 99%

“…In contrast, value score curves model the scaled economic value of a forecasting system for all economically rational decisions that users may face (Wilks 2001).…”

Section: Value Of Forecastsmentioning

confidence: 99%

“…Several standard verification methods, such as the Brier Skill Score (BSS) (Brier 1950;Toth et al 2003), the receiver operating characteristic (ROC) value (Mason 1982) and the reliability curve (Wilks 1995) are presented. The value of the forecasts for decisionmakers was estimated using value score curves (Wilks 2001). We also present the expected improvement in value scores associated with increased forecast variance and offer a new standard diagnostic verification parameter.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Verification and value of the Australian Bureau of Meteorology township seasonal rainfall forecasts in Australia, 1997–2005

Vizard

Anderson

Buckley

2005

Meteorological Applications

View full text Add to dashboard Cite

Application of decision‐making to a solar flare forecast in the cost‐loss ratio situation

et al. 2017

View full text Add to dashboard Cite

The conventional skill scores for evaluating flare forecast models do not take into account the effect of various cost‐loss ratios. For the first time, we have applied a decision‐making based on skill scores to a flare forecast model in cost‐loss ratio situations. For this study, we consider a flare forecast model which provides daily solar flare probabilities from 2011 to 2014. The skill scores are computed through contingency tables as a function of probability threshold (Pth) for the decision‐making. A value score (VS) is calculated as a function of cost‐loss ratio (C/L) and Pth, which are linearly correlated with each other. The maximum values of VS are 0.57, 0.37, and 0.61 for C‐, M‐, and X‐class flare, respectively. We expect that this study would provide a guideline to determine C/L and Pth for the better decision‐making in similar forecast models.

show abstract

A skill score based on economic value for probability forecasts

Cited by 122 publications

References 33 publications

On measuring quality of a probabilistic commodity forecast for a system that incorporates seasonal climate forecasts

On measuring quality of a probabilistic commodity forecast for a system that incorporates seasonal climate forecasts

Verification and value of the Australian Bureau of Meteorology township seasonal rainfall forecasts in Australia, 1997–2005

Application of decision‐making to a solar flare forecast in the cost‐loss ratio situation

Contact Info

Product

Resources

About