Forecast dominance testing via sign randomization

Ehm, Werner; Krüger, Fabian

doi:10.1214/18-ejs1495

Cited by 8 publications

(7 citation statements)

References 61 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hence the present time series example matches the setup of Equation (4). Example 2.1 of Ehm and Krüger (2018) is obtained as a special case if a A = a B = a Y and τ Y j = τ 2 j , j ∈ {A, B}, such that both forecasts are auto-calibrated. In the latter situation, the forecast j for which τ j is greater dominates its competitor.…”

Section: Forecast Dominance Under Normalitymentioning

confidence: 99%

Generic Conditions for Forecast Dominance

Krüger

Ziegel

2020

Journal of Business & Economic Statistics

Self Cite

View full text Add to dashboard Cite

Recent studies have analyzed whether one forecast method dominates another under a class of consistent scoring functions. While the existing literature focuses on empirical tests of forecast dominance, little is known about the theoretical conditions under which one forecast dominates another. To address this question, we first derive a new characterization of dominance among forecasts of the mean functional. We then present various scenarios under which dominance occurs. Unlike existing results, our results allow for the case that the forecasts' underlying information sets are not nested, and allow for uncalibrated forecasts that suffer, e.g., from model misspecification or parameter estimation error. We illustrate the empirical relevance of our results via data examples from finance, economics and meteorology.

show abstract

Section: Forecast Dominance Under Normalitymentioning

confidence: 99%

Generic Conditions for Forecast Dominance

Krüger

Ziegel

2020

Journal of Business & Economic Statistics

Self Cite

View full text Add to dashboard Cite

show abstract

“…If the observations are not independent, as usual in sequential settings, a number of asymptotic tests are available to compute p-values, with prominent ones being the Diebold-Mariano test (Diebold and Mariano, 1995) and the test of conditional predictive ability by Giacomini and White (Giacomini and White, 2006). Further examples are the martingale-based approaches by Seillier-Moiseiwitsch and Dawid (1993) or Lai et al (2011), and more recent tests of forecast dominance (Ehm and Krüger, 2018;Yen and Yen, 2021).…”

Section: Introductionmentioning

confidence: 99%

Valid sequential inference on probability forecast performance

Henzi¹,

Ziegel²

2021

Preprint

View full text Add to dashboard Cite

Probability forecasts for binary events play a central role in many applications. Their quality is commonly assessed with proper scoring rules, which assign forecasts a numerical score such that a correct forecast achieves a minimal expected score. In this paper, we construct e-values for testing the statistical significance of score differences of competing forecasts in sequential settings. E-values have been proposed as an alternative to p-values for hypothesis testing, and they can easily be transformed into conservative p-values by taking the multiplicative inverse. The e-values proposed in this article are valid in finite samples without any assumptions on the data generating processes. They also allow optional stopping, so a forecast user may decide to interrupt evaluation taking into account the available data at any time and still draw statistically valid inference, which is generally not true for classical p-value based tests. In a case study on postprocessing of precipitation forecasts, state-of-the-art forecasts dominance tests and e-values lead to the same conclusions.

show abstract

“…χ 2 (m: window size) (mixing) Lai et al (2011) H 0 : E[ δt | G t−1 ] = 0 √ t( ∆t − ∆ t )/s t N (0, 1) Ehm and Krüger (2018) H w 0 : 1…”

Section: Introductionmentioning

confidence: 99%

“…This distinction of strong and weak nulls come from the discussion of randomization experiments in the causal inference literature; see, e.g.,Lehmann (1975);Rosenbaum (1995);Wu and Ding (2020). Within the context of sequential forecast comparison,Ehm and Krüger (2018) makes the distinction between tests of average and step-by-step conditional predictive ability, which mirrors that of weak and strong nulls.…”

mentioning

confidence: 99%

Comparing Sequential Forecasters

Choe¹,

Ramdas²

2021

Preprint

View full text Add to dashboard Cite

Consider two or more forecasters, each making a sequence of predictions for different events over time. We ask a relatively basic question: how might we compare these forecasters, either online or post-hoc, while avoiding unverifiable assumptions on how the forecasts or outcomes were generated? This work presents a novel and rigorous answer to this question. We design a sequential inference procedure for estimating the time-varying difference in forecast quality as measured by a relatively large class of proper scoring rules (bounded scores with a linear equivalent). The resulting confidence intervals are nonasymptotically valid, and can be continuously monitored to yield statistically valid comparisons at arbitrary data-dependent stopping times ("anytime-valid"); this is enabled by adapting variance-adaptive supermartingales, confidence sequences, and e-processes to our setting. Motivated by Shafer and Vovk's game-theoretic probability, our coverage guarantees are also distribution-free, in the sense that they make no distributional assumptions on the forecasts or outcomes. In contrast to a recent work by Henzi and Ziegel, our tools can sequentially test a weak null hypothesis about whether one forecaster outperforms another on average over time. We demonstrate their effectiveness by comparing forecasts on Major League Baseball (MLB) games and statistical postprocessing methods for ensemble weather forecasts.

show abstract

Forecast dominance testing via sign randomization

Cited by 8 publications

References 61 publications

Generic Conditions for Forecast Dominance

Generic Conditions for Forecast Dominance

Valid sequential inference on probability forecast performance

Comparing Sequential Forecasters

Contact Info

Product

Resources

About