Linear Filtering of Sample Covariances for Ensemble-Based Data Assimilation. Part I: Optimality Criteria and Application to Variance Filtering and Covariance Localization

Abstract

In data assimilation (DA) schemes for numerical weather prediction (NWP) systems, the estimation of forecast error covariances is a key point to get some flow dependency. As shown in previous studies, ensemble data assimilation methods are the most accurate for this task. However, their huge computational cost raises a strong limitation to the ensemble size. Consequently, covariances estimated with small ensembles are affected by random sampling errors. The aim of this study is to develop a theory of covariance filtering in order to remove most of the sampling noise while keeping the signal of interest and then to use it in the DA scheme of a real NWP system. This first part of a two-part study presents the theoretical aspects of such criteria for optimal filtering based on the merging of the theories of optimal linear filtering and of sample centered moments estimation. Its strength relies on the use of sample estimated quantities and filter output only. These criteria pave the way for new algorithms and interesting applications for NWP. Two of them are detailed here: spatial filtering of variances and covariance localization. Results obtained in an idealized 1D analytical framework are shown for illustration. Applications on real forecast error covariances deduced from ensembles at convective scale are discussed in a companion paper.

Diagnosing non-Gaussianity of forecast and analysis errors in a convective-scale model

“…Those results agree with one of the conclusions of Ménétrier et al (2015). These authors describe an algorithm to find the optimal truncation dedicated to sample covariances filtering.…”

Accounting for model error in air quality forecasts: an application of 4DEnVar to the assimilation of atmospheric composition using QG-Chem 1.0

“…The development of more sophisticated localization operators (Bocquet, 2016;Desroziers et al, 2016) and more rigorous methods to estimate their parameters (Ménétrier et al, 2015), represents a current subject of research, and will be the topic of a future study.…”

Spectral diagonal ensemble Kalman filters

“…Formulations such as the diffusion operator or the recursive filter are related to the diagonal assumption here, they involve covariance models with a relatively small number of parameters and thus free of sampling noise but estimated from an ensemble directly (Pannekoucke and Massart, 2008;Michel, 2013;Pannekoucke et al, 2014). Similar filtering strategies can be employed to improve the estimation and the design of covariance formulations using results on the estimation of variances and length scales Raynaud et al, 2009;Raynaud and Pannekoucke, 2013;Ménétrier et al, 2015). The formulation of the background error covariance model using the diagonal assumption and a product of linear operator (such as the discrete Fourier or wavelet transform here) is widely used in variational literature to build covariance models in high dimension (e.g., Courtier et al, 1998;Fisher and Andersson, 2001;Weaver and Courtier, 2001).…”