Variogram-Based Proper Scoring Rules for Probabilistic Forecasts of Multivariate Quantities*

Scheuerer, Michael; Hamill, Thomas M.

doi:10.1175/mwr-d-14-00269.1

Cited by 153 publications

(207 citation statements)

References 40 publications

Supporting

Mentioning

195

Contrasting

Order By: Relevance

“…In a more general context of a full ensemble system of snowpack modelling with various applications, the selection of members might be improved by defining multiobjective probabilistic criteria combining several evaluation variables or even several evaluation sites. Recent investigations on that topic for the purpose of ensemble meteorological forecasting proposed generalisations of the classical univariate probabilistic tools (Gneiting et al, 2008;Scheuerer and Hamill, 2015;Thorarinsdottir et al, 2016) which could be tested in ensemble snow modelling. Special care should be taken in the future to deal with the covariance of errors among the different evaluation variables.…”

Section: Limitations Of Scores and Selection Methodsmentioning

confidence: 99%

A multiphysical ensemble system of numerical snow modelling

Lafaysse¹,

Cluzet²,

Dumont³

et al. 2017

The Cryosphere

103

135

View full text Add to dashboard Cite

Abstract. Physically based multilayer snowpack models suffer from various modelling errors. To represent these errors, we built the new multiphysical ensemble system ES-CROC (Ensemble System Crocus) by implementing new representations of different physical processes in the deterministic coupled multilayer ground/snowpack model SUR-FEX/ISBA/Crocus. This ensemble was driven and evaluated at Col de Porte (1325 m a.s.l., French alps) over 18 years with a high-quality meteorological and snow data set. A total number of 7776 simulations were evaluated separately, accounting for the uncertainties of evaluation data. The ability of the ensemble to capture the uncertainty associated to modelling errors is assessed for snow depth, snow water equivalent, bulk density, albedo and surface temperature. Different sub-ensembles of the ESCROC system were studied with probabilistic tools to compare their performance. Results show that optimal members of the ESCROC system are able to explain more than half of the total simulation errors. Integrating members with biases exceeding the range corresponding to observational uncertainty is necessary to obtain an optimal dispersion, but this issue can also be a consequence of the fact that meteorological forcing uncertainties were not accounted for. The ESCROC system promises the integration of numerical snow-modelling errors in ensemble forecasting and ensemble assimilation systems in support of avalanche hazard forecasting and other snowpack-modelling applications.

show abstract

Section: Limitations Of Scores and Selection Methodsmentioning

confidence: 99%

A multiphysical ensemble system of numerical snow modelling

Lafaysse¹,

Cluzet²,

Dumont³

et al. 2017

The Cryosphere

103

135

View full text Add to dashboard Cite

show abstract

“…The energy score reduces to the Euclidean error

‖ bold-italicμ - {bold-italicy}_{obs} ‖

for a deterministic point mass forecasts at μ and can be reported in the same unit as the observations. It has been noted that the ES discriminates well between forecasts with different means or variances, but less so for forecasts with different correlation structures.…”

Section: Predictive Model Assessmentmentioning

confidence: 99%

Probabilistic forecasting in infectious disease epidemiology: the 13th Armitage lecture

2017

View full text Add to dashboard Cite

Routine surveillance of notifiable infectious diseases gives rise to daily or weekly counts of reported cases stratified by region and age group. From a public health perspective, forecasts of infectious disease spread are of central importance. We argue that such forecasts need to properly incorporate the attached uncertainty, so they should be probabilistic in nature. However, forecasts also need to take into account temporal dependencies inherent to communicable diseases, spatial dynamics through human travel and social contact patterns between age groups. We describe a multivariate time series model for weekly surveillance counts on norovirus gastroenteritis from the 12 city districts of Berlin, in six age groups, from week 2011/27 to week 2015/26. The following year (2015/27 to 2016/26) is used to assess the quality of the predictions. Probabilistic forecasts of the total number of cases can be derived through Monte Carlo simulation, but first and second moments are also available analytically. Final size forecasts as well as multivariate forecasts of the total number of cases by age group, by district and by week are compared across different models of varying complexity. This leads to a more general discussion of issues regarding modelling, prediction and evaluation of public health surveillance data. Copyright © 2017 John Wiley & Sons, Ltd.

show abstract

“…Its analogue for multivariate quantities is the energy score (ES: Gneiting et al , ), which in the case of an ensemble forecast is computed via

\begin{matrix} alignleft \\ align-1 & align-2 ES (x_{1}, \dots, x_{N}; y) \\ align-1 & align-2 = \frac{1}{N} \sum_{n = 1}^{N} | | x_{n} - y | | - \frac{1}{2 N^{2}} \sum_{ν = 1}^{N} \sum_{n = 1}^{N} | | x_{ν} - x_{n} | |, \end{matrix}

where ||·|| denotes the Euclidean norm. While the ES is suitable to emphasize the benefits of postprocessing itself, it might fail to detect mis‐specifications in the correlation structures (Pinson and Tastu, ; Scheuerer and Hamill, ). In contrast, the variogram score (VS: Scheuerer and Hamill, ) is more sensitive in this respect.…”

Section: Case Studymentioning

confidence: 99%

Combining parametric low‐dimensional ensemble postprocessing with reordering methods

Schefzik

2016

Quart J Royal Meteoro Soc

View full text Add to dashboard Cite

State-of-the-art weather forecasts usually rely on ensemble prediction systems, accounting for the different sources of uncertainty. As ensembles are typically uncalibrated, they should get statistically postprocessed. Several multivariate ensemble postprocessing techniques that additionally consider spatial, inter-variable and/or temporal dependences have been developed. These can be roughly divided into two groups. The first group comprises parametric, mainly low-dimensional, approaches that are tailored to specific settings. The second group involves non-parametric reordering methods that impose a specific dependence template on univariately postprocessed forecasts and are suitable in any dimension. In this article, these different strategies are combined, with the aim of exploiting the benefits of both concepts. Specifically, a high-dimensional postprocessing problem is divided into multiple low-dimensional instances, each of which is postprocessed via a suitable multivariate parametric method. From each postprocessed low-dimensional distribution, a sample is drawn, which is then reordered according to the corresponding multidimensional rank structure of an appropriately chosen dependence template. In this context, different ranking concepts for multivariate settings are discussed. Finally, all reordered samples are aggregated to obtain the overall postprocessed ensemble. The new approach is applied to ensemble forecasts for temperature and wind speed at several locations from the European Centre for Medium-Range Weather Forecasts, using a recent bivariate ensemble model output statistics postprocessing technique and a reordering based on the raw ensemble forecasts similar to the ensemble copula coupling method. It shows good predictive skill and outperforms reference ensembles.

show abstract

Variogram-Based Proper Scoring Rules for Probabilistic Forecasts of Multivariate Quantities*

Cited by 153 publications

References 40 publications

A multiphysical ensemble system of numerical snow modelling

A multiphysical ensemble system of numerical snow modelling

Probabilistic forecasting in infectious disease epidemiology: the 13th Armitage lecture

Combining parametric low‐dimensional ensemble postprocessing with reordering methods

Contact Info

Product

Resources

About