Measuring forecast performance in the presence of observation error

Ferro, Christopher A. T.

doi:10.1002/qj.3115

Cited by 28 publications

(43 citation statements)

References 35 publications

(67 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A potential explanation arises from differences in the effect of having neglected uncertainty of the verification data. Numerous studies have pointed out the need to account for observation uncertainty in the verification of ensembles (e.g., Saetra et al, 2004;Yamaguchi et al, 2016;Ferro, 2017). The error of representativeness of 2 m temperature observations at stations is expected to be significantly larger than the analysis uncertainty of 850 hPa temperature.…”

Section: Resultsmentioning

confidence: 99%

On the probabilistic skill of dual‐resolution ensemble forecasts

Leutbecher

Bouallègue

2019

Quart J Royal Meteoro Soc

View full text Add to dashboard Cite

Increasing spatial resolution and increasing ensemble size both tend to improve the skill of ensemble forecasts. Due to computational constraints, a balance needs to be found in operational NWP. Here, we examine a scenario where ensembles are formed by pooling k lower‐resolution and m higher‐resolution members such that the overall computational cost is equal to the constraint. The approach is applied to medium‐range weather forecasts with ECMWF's Integrated Forecasting System using horizontal resolutions of 18, 29 and 45 km and ensemble sizes ranging from 8 to 254 members. The methodology is similar to the multi‐level Monte‐Carlo approach but does not use stochastic perturbations that are shared between members at different levels. Probabilistic skill is quantified for 850 hPa temperature verified against analyses and 2 m temperature verified against station observations. Generally, dual‐resolution ensembles with similar numbers of lower and higher‐resolution members provide the optimal configuration for 2 m temperature prediction. In contrast, single‐resolution ensembles appear to be the most skilful for 850 hPa temperature. The dual‐resolution ensembles are a special kind of multi‐model ensemble. An analytic model describing the skill of such a multi‐model ensemble is developed and its parameters are estimated from the actual verification statistics. The model is capable of describing the general differences in behaviour between 2 m temperature and 850 hPa temperature.

show abstract

Section: Resultsmentioning

confidence: 99%

On the probabilistic skill of dual‐resolution ensemble forecasts

Leutbecher

Bouallègue

2019

Quart J Royal Meteoro Soc

View full text Add to dashboard Cite

show abstract

“…, ; Bellprat et al. , ; Ferro, ). However, for this study, while absolute values of skill were dependent on the reference reanalysis, the forecast improvements gained by applying stochastic methods for model uncertainty estimation were more or less independent of that, providing robust indicators of skill increase.…”

Section: Discussionmentioning

confidence: 99%

Seasonal to annual ocean forecasting skill and the role of model and observational uncertainty

Juricke

MacLeod

Weisheimer

et al. 2018

Quart J Royal Meteoro Soc

View full text Add to dashboard Cite

Accurate forecasts of the ocean state and the estimation of forecast uncertainties are crucial when it comes to providing skilful seasonal predictions. In this study we analyse the predictive skill and reliability of the ocean component in a seasonal forecasting system. Furthermore, we assess the effects of accounting for model and observational uncertainties. Ensemble forcasts are carried out with an updated version of the ECMWF seasonal forecasting model System 4, with a forecast length of ten months, initialized every May between 1981 and 2010. We find that, for essential quantities such as sea surface temperature and upper ocean 300 m heat content, the ocean forecasts are generally underdispersive and skilful beyond the first month mainly in the Tropics and parts of the North Atlantic. The reference reanalysis used for the forecast evaluation considerably affects diagnostics of forecast skill and reliability, throughout the entire ten‐month forecasts but mostly during the first three months. Accounting for parametrization uncertainty by implementing stochastic parametrization perturbations has a positive impact on both reliability (from month 3 onwards) as well as forecast skill (from month 8 onwards). Skill improvements extend also to atmospheric variables such as 2 m temperature, mostly in the extratropical Pacific but also over the midlatitudes of the Americas. Hence, while model uncertainty impacts the skill of seasonal forecasts, observational uncertainty impacts our assessment of that skill. Future ocean model development should therefore aim not only to reduce model errors but to simultaneously assess and estimate uncertainties.

show abstract

“…The idea is to use climate models as a third-party source of information to infer the statistics of observational errors. The rationale behind this argument is simple: standard skill scores used in forecast verification (e.g., correlation, root mean square error, Brier score) are sensitive to errors in both the forecast and the verification data (87,88). If one particular observational verification product is corrupted with larger errors than other products, this observational product should systematically stand out compared to others, when inspecting the forecast skill scores of model predictions.…”

Section: Case 4 Evaluation Of Observational Productsmentioning

confidence: 99%

Climate Models as Guidance for the Design of Observing Systems: the Case of Polar Climate and Sea Ice Prediction

Massonnet

2019

Curr Clim Change Rep

View full text Add to dashboard Cite

The Arctic and Antarctic are among the regions most exposed to climate change, but ironically, they are also the ones for which the least observations are available. Climate models have been instrumental in completing the big picture. It is generally accepted that observations feed the development of climate models: parameterizations are designed based on empirically observed relationships, climate model predictions are initialized using observational products, and numerical simulations are evaluated given matching observational datasets. Recent research suggests that the opposite also holds: climate models can feed the development of polar observational networks by indicating the type, location, frequency, and timing of measurements that would be most useful for answering a specific scientific question. Here, we review the foundations of this emerging notion with five cases borrowed from the field of polar prediction with a focus on sea ice (sub-seasonal to centennial time scales). We suggest that climate models, besides their usual purposes, can be used to objectively prioritize future observational needsif, of course, the limitations of the realism of these models have been recognized. This idea, which has been already extensively exploited in the context of Numerical Weather Prediction, reinforces the notion that observations and models are two sides of the same coin rather than distinct conceptual entities.

show abstract

Measuring forecast performance in the presence of observation error

Cited by 28 publications

References 35 publications

On the probabilistic skill of dual‐resolution ensemble forecasts

On the probabilistic skill of dual‐resolution ensemble forecasts

Seasonal to annual ocean forecasting skill and the role of model and observational uncertainty

Climate Models as Guidance for the Design of Observing Systems: the Case of Polar Climate and Sea Ice Prediction

Contact Info

Product

Resources

About