Impact of observational error on the validation of ensemble prediction systems

Abstract

Ensemble prediction systems (EPSs) are usually validated under the assumption that the verifying observations are exact. In this paper, two methods are considered for taking observation errors into account. In the 'perturbed-ensemble' method, which has already been studied by other authors, the predicted ensemble elements are randomly perturbed in a way that is consistent with the assumed observation error. In the 'observational-probability' method, which is new, a verifying observation is considered as defining, together with the assumed associated error, a probability distribution. All standard scores for evaluation of EPSs (reliability diagram, Brier score, ranked probability score (RPS), continuous RPS (CRPS), relative-operating-characteristics (ROC) curve area), with the exception of the rank histogram, remain defined in this second method. In particular, the classical reliability-resolution decomposition of the Brier score, and of its extension to the RPS and CRPS, remain defined.Numerical simulations, partially supported by theoretical considerations, show that, with respect to the case when observation errors are ignored, the perturbed-ensemble method improves reliability, as well as the ROC score, while it has no significant impact on resolution, as measured by the Brier score. The observational-probability method, on the other hand, degrades reliability and the ROC score, but improves resolution.With respect to the 'real' performance of the system (i.e. the one that would be diagnosed if no error were present), reliability is unchanged in the perturbed-ensemble method, while resolution and the ROC score are degraded. The observational-probability method degrades reliability and the ROC score. As for resolution, an optimum value of the observational error is found, below which resolution is improved.Diagnostics performed on the operational EPS of the Canadian Meteorological Centre confirm the results of the simulations as to the consequences of ignoring observation errors, or on the contrary of taking them into account through either of the two methods. The significance of those various results is discussed. This article replaces a previously published version (Q.

Methodologies to characterize uncertainties in regional reanalyses

“…They show that rank histograms are highly sensitive to the inclusion of observation errors, whereas reliability diagrams are less sensitive. Candille and Talagrand (2008) introduce the "observational probability" method by defining observation uncertainty as a normal distribution which guarantees that the uncertainty is variable in both mean and spread. Santos and Ghelli (2012) extend the previous work and developed the "observed observational probability" method which includes uncertainty in the verification process to variables that are non-Gaussian distributed, in particular precipitation.…”

“…Using a reliability diagram (Hamill, 1997), the reliability, sharpness, and resolution of the ensembles (here: ensembles of regional reanalyses) are evaluated. Another diagram which is widely used is the relative operating characteristics (ROC) plot (Masson, 1982;Candille and Talagrand, 2008). Here, POD is plotted against POFD and it answers the question what the ability of the forecast is to discriminate between events and non-events.…”

“…Here, POD is plotted against POFD and it answers the question what the ability of the forecast is to discriminate between events and non-events. And finally, there are different skill scores which test the skill of the probabilistic information in varying ways, such as the Brier (skill) score (Brier, 1950;Candille and Talagrand, 2008), which relates to the user question: "What is the magnitude of the probability forecast errors?" or the (continuous) ranked probability score (Hersbach (2000) and references therein), which relates to the user question: "how well did the probability forecast predict the category that the observation fell into?…”

Inherent Bounds on Forecast Accuracy due to Observation Uncertainty Caused by Temporal Sampling

“…Bowler (2006) argues that a verification metric should not be affected by the quality of the observations network (i.e., given a perfect forecast, the use of an erroneous observation should still yield a perfect forecast). He argues that an approach such as that proposed by Candille and Talagrand (2005) describing the observation error as a probability density function will penalize a perfect forecast. Bowler (2008) subsequently used data assimilation-derived covariance estimates of the observations error to randomly perturb individual ensemble members.…”

“…Bowler (2008) subsequently used data assimilation-derived covariance estimates of the observations error to randomly perturb individual ensemble members. Santos and Ghelli (2012) extended the approach by Candille and Talagrand (2008) who considered empirical distributions to provide a measure of the spatial ''representativeness'' error. Koh et al (2012) also considered the temporal scaling at a point through the use of spectral analysis.…”