Coherent dispersion criteria for optimal experimental design

Dawid, A. Philip; Sebastiani, Paola

doi:10.1214/aos/1018031101

Cited by 128 publications

(90 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…is proper relative to P (Dawid and Sebastiani, 1999;. For our additive observation model (14,15), the scoring rule…”

Section: Numerical Predictandsmentioning

confidence: 99%

Measuring forecast performance in the presence of observation error

Ferro

2017

Quart J Royal Meteoro Soc

View full text Add to dashboard Cite

A new framework is introduced for measuring the performance of probability forecasts when the true value of the predictand is observed with error. In these circumstances, proper scoring rules favour good forecasts of observations rather than of truth and yield scores that vary with the quality of the observations. Proper scoring rules thus can favour forecasters who issue worse forecasts of the truth and can mask real changes in forecast performance if observation quality varies over time. Existing approaches to accounting for observation error provide unsatisfactory solutions to these two problems. A new class of ‘error‐corrected’ proper scoring rules is defined that solves both problems by producing unbiased estimates of the scores that would be obtained if the forecasts could be verified against the truth. A general method for constructing error‐corrected proper scoring rules is given for the case of categorical predictands, and error‐corrected versions of the Dawid–Sebastiani scoring rule are proposed for numerical predictands. The benefits of accounting for observation error in ensemble post‐processing and in forecast verification are illustrated in three data examples that include forecasts for the occurrence of tornadoes and of aircraft icing.

show abstract

“…is proper relative to P (Dawid and Sebastiani, 1999;. For our additive observation model (14,15), the scoring rule…”

Section: Numerical Predictandsmentioning

confidence: 99%

Measuring forecast performance in the presence of observation error

Ferro

2017

Quart J Royal Meteoro Soc

View full text Add to dashboard Cite

show abstract

“…On the other hand, there are scoring rules [e.g., the logarithmic score by Roulston and Smith (2002), applied to a multivariate probability density function] that are more sensitive to misspecified correlations, but require that the forecast is given in terms of a predictive density, and are thus not applicable in the important case of ensemble forecasts. Dawid and Sebastiani (1999) proposed some multivariate scoring rules that depend only on the mean vector m F and the covariance matrix S F of the predictive distribution F. A particularly appealing example is the scoring rule [hereafter referred to as the Dawid-Sebastiani score (DSS)]:…”

Section: Introductionmentioning

confidence: 99%

Variogram-Based Proper Scoring Rules for Probabilistic Forecasts of Multivariate Quantities*

Scheuerer

Hamill

2015

Monthly Weather Review

153

195

View full text Add to dashboard Cite

Proper scoring rules provide a theoretically principled framework for the quantitative assessment of the predictive performance of probabilistic forecasts. While a wide selection of such scoring rules for univariate quantities exists, there are only few scoring rules for multivariate quantities, and many of them require that forecasts are given in the form of a probability density function. The energy score, a multivariate generalization of the continuous ranked probability score, is the only commonly used score that is applicable in the important case of ensemble forecasts, where the multivariate predictive distribution is represented by a finite sample. Unfortunately, its ability to detect incorrectly specified correlations between the components of the multivariate quantity is somewhat limited. In this paper the authors present an alternative class of proper scoring rules based on the geostatistical concept of variograms. The sensitivity of these variogram-based scoring rules to incorrectly predicted means, variances, and correlations is studied in a number of examples with simulated observations and forecasts; they are shown to be distinctly more discriminative with respect to the correlation structure. This conclusion is confirmed in a case study with postprocessed wind speed forecasts at five wind park locations in Colorado.

show abstract

“…The scoring rules are preferred by [28] to check the predictions from mixed models which include random effects for multivariate modelling. There are three commonly used proper scoring rules: the logarithmic score (LS) [29], the Dawid-Sebastiani score (DSS) [30] and the ranked probability score (RPS) [31, 32]. LS is the negative log-likelihood evaluated at the actual observation; DSS is the standardised difference between the actual observation and predictive mean value plus a penalty of the predictive variance; RPS is the sum of the Brier scores [33] for binary predictions at all possible thresholds.…”

Section: Methodsmentioning

confidence: 99%

Methodological challenges to multivariate syndromic surveillance: a case study using Swiss animal health data

2016

View full text Add to dashboard Cite

BackgroundIn an era of ubiquitous electronic collection of animal health data, multivariate surveillance systems (which concurrently monitor several data streams) should have a greater probability of detecting disease events than univariate systems. However, despite their limitations, univariate aberration detection algorithms are used in most active syndromic surveillance (SyS) systems because of their ease of application and interpretation. On the other hand, a stochastic modelling-based approach to multivariate surveillance offers more flexibility, allowing for the retention of historical outbreaks, for overdispersion and for non-stationarity. While such methods are not new, they are yet to be applied to animal health surveillance data. We applied an example of such stochastic model, Held and colleagues’ two-component model, to two multivariate animal health datasets from Switzerland.ResultsIn our first application, multivariate time series of the number of laboratories test requests were derived from Swiss animal diagnostic laboratories. We compare the performance of the two-component model to parallel monitoring using an improved Farrington algorithm and found both methods yield a satisfactorily low false alarm rate. However, the calibration test of the two-component model on the one-step ahead predictions proved satisfactory, making such an approach suitable for outbreak prediction. In our second application, the two-component model was applied to the multivariate time series of the number of cattle abortions and the number of test requests for bovine viral diarrhea (a disease that often results in abortions). We found that there is a two days lagged effect from the number of abortions to the number of test requests. We further compared the joint modelling and univariate modelling of the number of laboratory test requests time series. The joint modelling approach showed evidence of superiority in terms of forecasting abilities.ConclusionsStochastic modelling approaches offer the potential to address more realistic surveillance scenarios through, for example, the inclusion of times series specific parameters, or of covariates known to have an impact on syndrome counts. Nevertheless, many methodological challenges to multivariate surveillance of animal SyS data still remain. Deciding on the amount of corroboration among data streams that is required to escalate into an alert is not a trivial task given the sparse data on the events under consideration (e.g. disease outbreaks).

show abstract

Coherent dispersion criteria for optimal experimental design

Cited by 128 publications

References 28 publications

Measuring forecast performance in the presence of observation error

Measuring forecast performance in the presence of observation error

Variogram-Based Proper Scoring Rules for Probabilistic Forecasts of Multivariate Quantities*

Methodological challenges to multivariate syndromic surveillance: a case study using Swiss animal health data

Contact Info

Product

Resources

About