Inferential, Nonparametric Statistics to Assess the Quality of Probabilistic Forecast Systems

Abstract

Many statistical forecast systems are available to interested users. To be useful for decision making, these systems must be based on evidence of underlying mechanisms. Once causal connections between the mechanism and its statistical manifestation have been firmly established, the forecasts must also provide some quantitative evidence of "quality." However, the quality of statistical climate forecast systems (forecast quality) is an ill-defined and frequently misunderstood property. Often, providers and users of such forecast systems are unclear about what quality entails and how to measure it, leading to confusion and misinformation. A generic framework is presented that quantifies aspects of forecast quality using an inferential approach to calculate nominal significance levels ( p values), which can be obtained either by directly applying nonparametric statistical tests such as Kruskal-Wallis (KW) or Kolmogorov-Smirnov (KS) or by using Monte Carlo methods (in the case of forecast skill scores). Once converted to p values, these forecast quality measures provide a means to objectively evaluate and compare temporal and spatial patterns of forecast quality across datasets and forecast systems. The analysis demonstrates the importance of providing p values rather than adopting some arbitrarily chosen significance levels such as 0.05 or 0.01, which is still common practice. This is illustrated by applying nonparametric tests (such as KW and KS) and skill scoring methods [linear error in the probability space (LEPS) and ranked probability skill score (RPSS)] to the five-phase Southern Oscillation index classification system using historical rainfall data from Australia, South Africa, and India. The selection of quality measures is solely based on their common use and does not constitute endorsement. It is found that nonparametric statistical tests can be adequate proxies for skill measures such as LEPS or RPSS. The framework can be implemented anywhere, regardless of dataset, forecast system, or quality measure. Eventually such inferential evidence should be complemented by descriptive statistical methods in order to fully assist in operational risk management.

Forecasting rainfall based on the Southern Oscillation Index phases at longer lead-times in Australia

“…5). We found P-value maps derived from the hindcast KW and LEPS tests to be similar, a finding supported by Maia et al (2007), but there were differences between hindcast and forecast modes. Spatial forecast quality in hindcast and forecast modes was evident for large areas of north-eastern Australia (lead-times of 0-5 months) and the Pilbara region in Western Australia (0-3 months).…”

“…The quality of a forecast in a particular region may be useful and real even if it does not reach an arbitrary significance level of P = 0.01, 0.05, or 0.10. In this paper, actual P-values are reported (Nicholls 2001;Maia et al 2007).…”

Using multi-parameters distributions to assess the probability of occurrence of extreme rainfall data

“…As one may note (Table 1) all acf coefficients remained within the white noise limits. Thus, it may be assumed that using the parametric distributions to assess the probability of occurrence of the Pre-extrem data will result in loss of no significant information due to the presence of serial correlation (Maia et al, 2007). This lack of temporal persistence also allowed us to apply the MK test in its original form (Önöz & Bayazit, 2011, among many others).…”

“…According to Maia et al (2007), fitting a parametric distribution to a given data set is only appropriate for uncorrelated process. Thus, the auto-correlation function (acf) was used to verify if the data sample is free from such a component.…”

Aplicação do conceito do índice padronizado de precipitação à série decendial da diferença entre precipitação pluvial e evapotranspiração potencial

“…Conforme indicado por Maia et al (2007), a parametrização de uma amostra de dados é adequada somente a séries livres de correlação serial. A parametrização de séries com significativa persistência temporal acarreta em perda de importantes informações, pois as probabilidades associadas a eventos temporalmente subsequentes não podem ser consideradas independentes entre si.…”