Minimum Scoring Rule Inference

Summary. In the practice of point prediction, it is desirable that forecasters receive a directive in the form of a statistical functional. For example, forecasters might be asked to report the mean or a quantile of their predictive distributions. When evaluating and comparing competing forecasts, it is then critical that the scoring function used for these purposes be consistent for the functional at hand, in the sense that the expected score is minimized when following the directive. We show that any scoring function that is consistent for a quantile or an expectile functional can be represented as a mixture of elementary or extremal scoring functions that form a linearly parameterized family. Scoring functions for the mean value and probability forecasts of binary events constitute important examples. The extremal scoring functions admit appealing economic interpretations of quantiles and expectiles in the context of betting and investment problems. The Choquet-type mixture representations give rise to simple checks of whether a forecast dominates another in the sense that it is preferable under any consistent scoring function. In empirical settings it suffices to compare the average scores for only a finite number of extremal elements. Plots of the average scores with respect to the extremal scoring functions, which we call Murphy diagrams, permit detailed comparisons of the relative merits of competing forecasts.

show abstract

“…Now (see Dawid et al . ()), given any proper scoring rule

S^{*}

, and a parametric family

F = {F_{italicθ}}

, we can construct an associated M ‐estimator of θ . For independent identically distributed data

(y_{1}, \dots, y_{n})

, this is given by

\overset{false^}{italicθ} : = arg min_{italicθ} S^{*} false(F_{θ}, {\overset{false^}{F}}_{n} false),

where

{\overset{false^}{F}}_{n}

is the empirical distribution function of the data.…”

Section: Discussion On the Paper By Ehm Gneiting Jordan And Krügermentioning

confidence: 99%

“…Now (see Dawid et al (2016)), given any proper scoring rule S Å , and a parametric family F = {F θ }, we can construct an associated M-estimator of θ. For independent identically distributed data .y 1 , : : : , y n /, this is given byθ…”

Section: Monica Musio (Università Degli Studi DI Cagliari)mentioning

confidence: 99%

Of Quantiles and Expectiles: Consistent Scoring Functions, Choquet Representations and Forecast Rankings

Ehm

Gneiting

Jordan³

et al. 2016

Journal of the Royal Statistical Society Series B: Statistical Methodology

144

201

View full text Add to dashboard Cite

show abstract

“…Here,

m = 12

is the number of ensemble members and the coefficients

a

b

c

and

d

are real numbers. For estimating

a

b

c

and

d

, we use minimum score estimation (Dawid et al, ) and optimize the continuous ranked probability score (CRPS; Matheson and Winkler, ; Gneiting and Raftery, ) based on training data as suggested by Gneiting et al, (). Gebetsberger et al, () gives a comprehensive comparison of minimum CRPS and maximum likelihood estimation.…”

Section: Data and Conventional Post‐processingmentioning

confidence: 99%

Rapid adjustment and post‐processing of temperature forecast trajectories

Schuhen

Thorarinsdottir

Lenkoski

2020

Quart J Royal Meteoro Soc

View full text Add to dashboard Cite

Modern weather forecasts are commonly issued as consistent multi‐day forecast trajectories with a time resolution of 1–3 hours. Prior to issuing, statistical post‐processing is routinely used to correct systematic errors and misrepresentations of the forecast uncertainty. However, once the forecast has been issued, it is rarely updated before it is replaced in the next forecast cycle of the numerical weather prediction (NWP) model. This paper shows that the error correlation structure within the forecast trajectory can be utilized to substantially improve the forecast between the NWP forecast cycles by applying additional post‐processing steps each time new observations become available. The proposed rapid adjustment is applied to temperature forecast trajectories from the UK Met Office's convective‐scale ensemble MOGREPS‐UK. MOGREPS‐UK is run four times daily and produces hourly forecasts for up to 36 hours ahead. Our results indicate that the rapidly adjusted forecast from the previous NWP forecast cycle outperforms the new forecast for the first few hours of the next cycle, or until the new forecast itself can be rapidly adjusted, suggesting a new strategy for updating the forecast cycle.

show abstract

“…The choice = 2.5 provides an outlier-robust measure. 54 The Tsallis score can be used to compare different BNP models that differ by included covariates, prior distributions, and so forth.…”

Section: How To Cite This Articlementioning

confidence: 99%

A Bayesian nonparametric test of significance chasing biases

Karabatsos

2018

Research Synthesis Methods

View full text Add to dashboard Cite

There is a growing concern that much of the published research literature is distorted by the pursuit of statistically significant results. In a seminal article, Ioannidis and Trikalinos (2007, Clinical Trials) proposed an omnibus (I&T) test for significance chasing (SC) biases. This test compares the observed number of studies that report statistically significant results, against their expected number based on study power, assuming a common effect size across studies. The current article extends this approach by developing a Bayesian nonparametric (BNP) meta-regression model and test of SC bias, which can diagnose bias at the individual study level. This new BNP test is based on a flexible model of the predictive distribution of study power, conditionally on study-level covariates which account for study diversity, including diversity due to heterogeneous effect sizes across studies. A test of SC bias proceeds by comparing each study's significant outcome report indicator against its estimated posterior predictive distribution of study power, conditionally on the study's covariates. The BNP model and SC bias test are illustrated through the analyses of 3 meta-analytic data sets and through a simulation study. Software code for the BNP model and test, and the data sets, are provided as Supporting Information.

show abstract

Minimum Scoring Rule Inference

Cited by 51 publications

References 37 publications

Of Quantiles and Expectiles: Consistent Scoring Functions, Choquet Representations and Forecast Rankings

Of Quantiles and Expectiles: Consistent Scoring Functions, Choquet Representations and Forecast Rankings

Rapid adjustment and post‐processing of temperature forecast trajectories

A Bayesian nonparametric test of significance chasing biases

Contact Info

Product

Resources

About