Understanding forecast verification statistics

Mason, Simon J.

doi:10.1002/met.51

Cited by 73 publications

(70 citation statements)

References 58 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is a rather strong assumption given the little amount of data concerning carbon exchange rates and the lack of repeated measurements that are needed to define variability in a climatological sense. It is more likely to get good scores by accident with limited set of data (Mason, 2008), such as the monthly observations in the HOT and BATS timeseries.…”

Section: Discussion and Applicationsmentioning

confidence: 99%

“…The empirical distribution of score values was constructed with 10 000 random re-samplings of the observation (or simulation) time series and computing the verification index for each new set of model-data pairs. As pointed out by Mason (2008), p-values do not answer the question whether the score value is good, but rather they provide a degree of significance with respect to random combinations. The confidence interval is instead computed by means of the bootstrap technique, in which the choice of randomly permuted model-data pairs is done by replacing the extracted pairs in the original time series.…”

Section: Data Sets and Skill Indicatorsmentioning

confidence: 98%

“…The probability of obtaining a score value better than the one achieved is generally termed p-value (Mason, 2008). The empirical distribution of score values was constructed with 10 000 random re-samplings of the observation (or simulation) time series and computing the verification index for each new set of model-data pairs.…”

Section: Data Sets and Skill Indicatorsmentioning

confidence: 99%

See 2 more Smart Citations

Skill assessment of the PELAGOS global ocean biogeochemistry model over the period 1980–2000

Vichi

Masina

2009

Biogeosciences

View full text Add to dashboard Cite

Abstract. Global Ocean Biogeochemistry General Circulation Models are useful tools to study biogeochemical processes at global and large scales under current climate and future scenario conditions. The credibility of future estimates is however dependent on the model skill in capturing the observed multi-annual variability of firstly the mean bulk biogeochemical properties, and secondly the rates at which organic matter is processed within the food web. For this double purpose, the results of a multi-annual simulation of the global ocean biogeochemical model PELAGOS have been objectively compared with multi-variate observations from the last 20 years of the 20th century, both considering bulk variables and carbon production/consumption rates. Simulated net primary production (NPP) is comparable with satellite-derived estimates at the global scale and when compared with an independent data-set of in situ observations in the equatorial Pacific. The usage of objective skill indicators allowed us to demonstrate the importance of comparing like with like when considering carbon transformation processes. NPP scores improve substantially when in situ data are compared with modeled NPP which takes into account the excretion of freshly-produced dissolved organic carbon (DOC). It is thus recommended that DOC measurements be performed during in situ NPP measurements to quantify the actual production of organic carbon in the surface ocean. The chlorophyll bias in the Southern Ocean that affects this model as well as several others is linked to the inadequate representation of the mixed layer seasonal cycle in the region. A sensitivity experiment confirms that the artificial increase of mixed layer depths towards the observed values substantially reduces the bias. Our assessment resultsCorrespondence to: M. Vichi (vichi@bo.ingv.it) qualify the model for studies of carbon transformation in the surface ocean and metabolic balances. Within the limits of the model assumption and known biases, PELAGOS indicates a net heterotrophic balance especially in the more oligotrophic regions of the Atlantic during the boreal winter period. However, at the annual time scale and over the global ocean, the model suggests that the surface ocean is close to a weakly positive autotrophic balance in accordance with recent experimental findings and geochemical considerations.

show abstract

Section: Discussion and Applicationsmentioning

confidence: 99%

Section: Data Sets and Skill Indicatorsmentioning

confidence: 98%

Section: Data Sets and Skill Indicatorsmentioning

confidence: 99%

See 1 more Smart Citation

Skill assessment of the PELAGOS global ocean biogeochemistry model over the period 1980–2000

Vichi

Masina

2009

Biogeosciences

View full text Add to dashboard Cite

show abstract

“…The reference forecasts are generated using a permutation procedure (see e.g. Mason, 2008;Deque, 2012). The permutation procedure generates a new set of forecasts-observation pairs in which observation are unrelated to the forecasts except by chance.…”

Section: Variation Of Evaluation Scores With Forecast Lead Timesmentioning

confidence: 99%

Evaluation of numerical weather prediction model precipitation forecasts for short-term streamflow forecasting purpose

Shrestha

Robertson

Wang

et al. 2013

Hydrol. Earth Syst. Sci.

117

View full text Add to dashboard Cite

Abstract. The quality of precipitation forecasts from four Numerical Weather Prediction (NWP) models is evaluated over the Ovens catchment in Southeast Australia. Precipitation forecasts are compared with observed precipitation at point and catchment scales and at different temporal resolutions. The four models evaluated are the Australian Community Climate Earth-System Simulator (ACCESS) including ACCESS-G with a 80 km resolution, ACCESS-R 37.5 km, ACCESS-A 12 km, and ACCESS-VT 5 km.The skill of the NWP precipitation forecasts varies considerably between rain gauging stations. In general, high spatial resolution (ACCESS-A and ACCESS-VT) and regional (ACCESS-R) NWP models overestimate precipitation in dry, low elevation areas and underestimate in wet, high elevation areas. The global model (ACCESS-G) consistently underestimates the precipitation at all stations and the bias increases with station elevation. The skill varies with forecast lead time and, in general, it decreases with the increasing lead time. When evaluated at finer spatial and temporal resolution (e.g. 5 km, hourly), the precipitation forecasts appear to have very little skill. There is moderate skill at short lead times when the forecasts are averaged up to daily and/or catchment scale. The precipitation forecasts fail to produce a diurnal cycle shown in observed precipitation. Significant sampling uncertainty in the skill scores suggests that more data are required to get a reliable evaluation of the forecasts. The non-smooth decay of skill with forecast lead time can be attributed to diurnal cycle in the observation and sampling uncertainty.Future work is planned to assess the benefits of using the NWP rainfall forecasts for short-term streamflow forecasting. Our findings here suggest that it is necessary to remove the systematic biases in rainfall forecasts, particularly those from low resolution models, before the rainfall forecasts can be used for streamflow forecasting.

show abstract

“…Bradley et al (2008) derive analytical expressions for the sampling variances of both the Brier score and the Brier skill score with respect to the sample climatological baseline. In broader overviews, Jolliffe (2007) reviews concepts in statistical inference as applicable to forecast verification, and Mason (2008) discusses the necessity of statistical inference on verification scores in order to evaluate whether a given result reflects meaningful skill above a naive baseline.…”

Section: Introductionmentioning

confidence: 99%

Sampling distributions of the Brier score and Brier skill score under serial dependence

Wilks

2010

Quart J Royal Meteoro Soc

View full text Add to dashboard Cite

Relatively little attention has been given to the effects of serial correlation of forecasts and observations on the sampling properties of forecast verification statistics. An assumption of serial independence for low-quality forecasts may be reasonable. However, forecasts of sufficient quality for autocorrelated events must themselves be autocorrelated: as quality approaches the limit of perfect forecasts, the forecasts become increasingly similar to the corresponding observations.The effects of forecast serial correlation on the sampling properties of the Brier Score (BS) and Brier Skill Score (BSS), for probability forecasts of dichotomous events, are examined here. As in other settings, the effect of serial correlation is to inflate the variances of the sampling distributions of the two statistics, so that uncorrected confidence intervals are too narrow, and uncorrected hypothesis tests yield p-values that are too small. Expressions are given for 'effective sample size' corrections for the sampling variances of both BS and BSS, in which it can be seen that the effects of serial correlation on the sampling variances increase with increasing forecast accuracy, and with decreasing climatological event probability. The sampling variance of BSS is more robust to serial correlation than that of BS. Hypothesis tests based on BSS are seen to be more powerful (i.e. more sensitive) than those based on BS, and substantially so for lower-accuracy forecasts of lowerprobability events, for both serially correlated and temporally independent forecasts.

show abstract

Understanding forecast verification statistics

Cited by 73 publications

References 58 publications

Skill assessment of the PELAGOS global ocean biogeochemistry model over the period 1980–2000

Skill assessment of the PELAGOS global ocean biogeochemistry model over the period 1980–2000

Evaluation of numerical weather prediction model precipitation forecasts for short-term streamflow forecasting purpose

Sampling distributions of the Brier score and Brier skill score under serial dependence

Contact Info

Product

Resources

About