Scoring Rules for Forecast Verification

Benedetti, Riccardo

doi:10.1175/2009mwr2945.1

Cited by 79 publications

(63 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This was achieved by use of a Taylor series expansion to write a logarithmic scoring rule in terms of a quadratic approximation. More recently, Benedetti [24] has attributed the lasting application of the Brier score in forecast evaluation to its being an approximation of the logarithmic score; however, an analysis leading to the Brier score as an approximation of the logarithmic score does not reveal a hierarchy in which the latter is in some way more fundamental than the former (cf. [25]).…”

Section: Overviewmentioning

confidence: 99%

See 1 more Smart Citation

Probabilistic Forecasts: Scoring Rules and Their Decomposition and Diagrammatic Representation via Bregman Divergences

Hughes

Topp

2015

Entropy

View full text Add to dashboard Cite

Abstract:A scoring rule is a device for evaluation of forecasts that are given in terms of the probability of an event. In this article we will restrict our attention to binary forecasts. We may think of a scoring rule as a penalty attached to a forecast after the event has been observed. Thus a relatively small penalty will accrue if a high probability forecast that an event will occur is followed by occurrence of the event. On the other hand, a relatively large penalty will accrue if this forecast is followed by non-occurrence of the event.Meteorologists have been foremost in developing scoring rules for the evaluation of probabilistic forecasts. Here we use a published meteorological data set to illustrate diagrammatically the Brier score and the divergence score, and their statistical decompositions, as examples of Bregman divergences. In writing this article, we have in mind environmental scientists and modellers for whom meteorological factors are important drivers of biological, physical and chemical processes of interest. In this context, we briefly draw attention to the potential for probabilistic forecasting of the within-season component of nitrous oxide emissions from agricultural soils.

show abstract

Section: Overviewmentioning

confidence: 99%

“…On this basis, neither scoring rule is inherently superior to the other. However, it is possible to establish further criteria against which the properties of such scoring rules may be judged [24]. The statistical decomposition of the scoring rule in Equation (4) also has a common format:…”

Section: Overviewmentioning

confidence: 99%

Probabilistic Forecasts: Scoring Rules and Their Decomposition and Diagrammatic Representation via Bregman Divergences

Hughes

Topp

2015

Entropy

View full text Add to dashboard Cite

show abstract

“…forecast probabilities of 100 % or 0 %) (Jewson, 2008;Benedetti, 2010). We adopt the recommendations of Benedetti (2010) and Jewson (2008), who both advocate variations on the likelihood to assess probabilistic forecasts. We term this measure the log-likelihood ratio (LLR).…”

Section: Log-likelihood Ratiomentioning

confidence: 99%

“…Brier scores show more instances of positive skill than LLR scores, particularly for streamflows larger than Q 10 . Because the Brier score has known problems with infrequent events (Benedetti, 2010), we focus on the LLR score to discuss forecast skill at larger streamflows. Substantial skill is evident in forecasts where observed Max1D streamflows are larger than Q 50 for 1-month forecasts, in both the Brier score (Fig.…”

Section: Forecast Skill For Large Threshold Eventsmentioning

confidence: 99%

The challenge of forecasting high streamflows 1–3 months in advance with lagged climate indices in southeast Australia

Bennett

Wang

Pokhrel

et al. 2014

Nat. Hazards Earth Syst. Sci.

View full text Add to dashboard Cite

Abstract. Skilful forecasts of high streamflows a month or more in advance are likely to be of considerable benefit to emergency services and the broader community. This is particularly true for mesoscale catchments (< 2000 km 2 ) with little or no seasonal snowmelt, where real-time warning systems are only able to give short notice of impending floods. In this study, we generate forecasts of high streamflows for the coming 1-month and coming 3-month periods using large-scale ocean-atmosphere climate indices and catchment wetness as predictors. Forecasts are generated with a combination of Bayesian joint probability modelling and Bayesian model averaging. High streamflows are defined as maximum single-day streamflows and maximum 5-day streamflows that occur during each 1-month or 3-month forecast period. Skill is clearly evident in the 1-month forecasts of high streamflows. Surprisingly, in several catchments positive skill is also evident in forecasts of large threshold events (exceedance probabilities of 25 %) over the next month. Little skill is evident in forecasts of high streamflows for the 3-month period. We show that including lagged climate indices as predictors adds little skill to the forecasts, and thus catchment wetness is by far the most important predictor. Accordingly, we recommend that forecasts may be improved by using accurate estimates of catchment wetness.

show abstract

“…Some authors suggested that "locality" is a desirable characteristic for a scoring rule (e.g. Bickel, 2007;Benedetti, 2010). There is no perfect value to use as a reference for the logarithmic score and no deterministic counterpart.…”

Section: Evaluation Of Performancementioning

confidence: 99%

A comparison between ensemble and deterministic hydrological forecasts in an operational context

Boucher¹,

Anctil²,

Perreault³

et al. 2011

Adv. Geosci.

View full text Add to dashboard Cite

Abstract. Ensemble forecasts can greatly benefit water resources management as they provide useful information regarding the uncertainty of the situation at hand. However, weather forecasting systems are evolving and the cost for reanalysis and reforecasts is prohibitive. Consequently, series of ensemble weather forecasts from a particular version of the forecasting system are often short. In this case study, we consider a hydrological event that took place in 2003 on the Gatineau watershed in Canada and caused management difficulties in a hydropower production context. The weather ensemble forecasting system in place at that time is now obsolete, but we show that with minimal post-processing of the forecasts, it is still beneficial to exploit ensemble rather than deterministic forecasts, even if the latter emerge from a more advanced meteorological model and possess superior spatial resolution.

show abstract

Scoring Rules for Forecast Verification

Cited by 79 publications

References 14 publications

Probabilistic Forecasts: Scoring Rules and Their Decomposition and Diagrammatic Representation via Bregman Divergences

Probabilistic Forecasts: Scoring Rules and Their Decomposition and Diagrammatic Representation via Bregman Divergences

The challenge of forecasting high streamflows 1–3 months in advance with lagged climate indices in southeast Australia

A comparison between ensemble and deterministic hydrological forecasts in an operational context

Contact Info

Product

Resources

About