The impact of using reconditioned correlated observation‐error covariance matrices in the Met Office 1D‐Var system

Tabeart, Jemima M.; Dance, Sarah L.; Lawless, Amos S.; Migliorini, Stefano; Nichols, Nancy K.; Smith, F. B.; Waller, John C.

doi:10.1002/qj.3741

Cited by 13 publications

(17 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the 4D‐Var problem the generalized observation operator

\hat{H}

also accounts for model evolution, and hence the structure of the linearized model is also expected to be important when considering clustering and convergence of a conjugate gradient problem. Previous work has also shown that for the unpreconditioned problem, the qualitative behavior of an operational system 25 largely followed the linear theory 27 . Similarly, for the case of uncorrelated OEC matrices, the behavior of preconditioned 4D‐Var experiments broadly coincided with theory from the linear setting 14,26 .…”

Section: Discussionmentioning

confidence: 62%

“…Correlated OEC matrices also lead to greater information content of observations, particularly on smaller scales 19,21‐23 . However, the move from uncorrelated (diagonal) to correlated (full) covariance matrices has caused problems with the convergence of the data assimilation procedure in experiments at NWP centers 17,24,25 . Previous studies of the conditioning of the preconditioned Hessian have focused on the case of uncorrelated OEC matrices 14,26 .…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

New bounds on the condition number of the Hessian of the preconditioned variational data assimilation problem

Tabeart

Dance

Lawless

et al. 2021

Numerical Linear Algebra App

Self Cite

View full text Add to dashboard Cite

Data assimilation algorithms combine prior and observational information, weighted by their respective uncertainties, to obtain the most likely posterior of a dynamical system. In variational data assimilation the posterior is computed by solving a nonlinear least squares problem. Many numerical weather prediction (NWP) centers use full observation error covariance (OEC) weighting matrices, which can slow convergence of the data assimilation procedure. Previous work revealed the importance of the minimum eigenvalue of the OEC matrix for conditioning and convergence of the unpreconditioned data assimilation problem. In this article we examine the use of correlated OEC matrices in the preconditioned data assimilation problem for the first time. We consider the case where there are more state variables than observations, which is typical for applications with sparse measurements, for example, NWP and remote sensing. We find that similarly to the unpreconditioned problem, the minimum eigenvalue of the OEC matrix appears in new bounds on the condition number of the Hessian of the preconditioned objective function. Numerical experiments reveal that the condition number of the Hessian is minimized when the background and observation lengthscales are equal. This contrasts with the unpreconditioned case, where decreasing the observation error lengthscale always improves conditioning. Conjugate gradient experiments show that in this framework the condition number of the Hessian is a good proxy for convergence. Eigenvalue clustering explains cases where convergence is faster than expected.

show abstract

“…For the 4D‐Var problem the generalized observation operator

\hat{H}

Section: Discussionmentioning

confidence: 62%

Section: Introductionmentioning

confidence: 99%

New bounds on the condition number of the Hessian of the preconditioned variational data assimilation problem

Tabeart

Dance

Lawless

et al. 2021

Numerical Linear Algebra App

Self Cite

View full text Add to dashboard Cite

show abstract

“…Correlated OEC matrices also lead to greater information content of observations, particularly on smaller scales 12,38,41,42 . However, the move from uncorrelated (diagonal) to correlated (full) covariance matrices, has been shown to cause problems with the convergence of the data assimilation procedure in experiments at NWP centres 44,51,52 . Previous studies of the conditioning of the preconditioned Hessian have focussed on the case of uncorrelated OEC matrices 18,19 .…”

Section: Introductionmentioning

confidence: 99%

The conditioning of least‐squares problems in variational data assimilation

Tabeart

Dance

Haben

et al. 2018

Numerical Linear Algebra App

Self Cite

103

View full text Add to dashboard Cite

SummaryIn variational data assimilation a least-squares objective function is minimised to obtain the most likely state of a dynamical system. This objective function combines observation and prior (or background) data weighted by their respective error statistics. In numerical weather prediction, data assimilation is used to estimate the current atmospheric state, which then serves as an initial condition for a forecast. New developments in the treatment of observation uncertainties have recently been shown to cause convergence problems for this least-squares minimisation. This is important for operational numerical weather prediction centres due to the time constraints of producing regular forecasts. The condition number of the Hessian of the objective function can be used as a proxy to investigate the speed of convergence of the least-squares minimisation. In this paper we develop novel theoretical bounds on the condition number of the Hessian. These new bounds depend on the minimum eigenvalue of the observation error covariance matrix and the ratio of background error variance to observation error variance. Numerical tests in a linear setting show that the location of observation measurements has an important effect on the condition number of the Hessian.We identify that the conditioning of the problem is related to the complex interactions between observation error covariance and background error covariance matrices. Increased understanding of the role of each constituent matrix in the conditioning of the Hessian will prove useful for informing the choice of correlated observation error covariance matrix and observation location, particularly for practical applications.

show abstract

“…This reduction is due to the fact that the diagonal matrix is pulling much closer to the observations than the correlated matrix, which makes it harder to find a solution resulting in slower convergence. This increase of the convergence speed was encountered in the Met Office 1D-Var system (Tabeart et al, 2020a) where a correlated observation matrix was introduced in the system. Furthermore, in Tabeart et al (2018) the matrix R and the observation error variance appear in the expression of the condition number of the Hessian of the variational assimilation problem, indicating that these terms are important for convergence of the minimization function.…”

Section: Computational Costmentioning

confidence: 97%

Estimation of the error covariance matrix for IASI radiances and its impact on ozone analyses

Aabaribaoune¹,

Emili²,

Guidard³

2020

Preprint

View full text Add to dashboard Cite

Abstract. In atmospheric chemistry retrievals and data assimilation systems, observation errors associated with satellite radiances are chosen empirically and generally treated as uncorrelated. In this work, we estimate inter-channel error covariances for the Infrared Atmospheric Sounding Interferometer (IASI) and evaluate their impact on ozone assimilation with the chemical transport model MOCAGE (MOdèle de Chime Atmospheric à Grand Echelle). The method used to calculate observation errors is a diagnostic based on the observation and analysis residual statistics already adopted in numerical weather prediction centers. We used a subset of 280 channels covering the spectral range between 980 and 1100 cm−1 to estimate the observation error covariance matrix. We computed hourly 3D-Var analyses and compared the resulting O3 fields against ozonesondes and the measurements provided by the Microwave Limb Sounder (MLS). The results show significant differences between using the estimated error covariance matrix with respect to the empirical diagonal matrix employed in previous studies. The validation of the analyses against independent data reports a significant improvement especially in the tropical stratosphere. The computational cost has also been reduced when the estimated covariance is employed in the assimilation system.

show abstract

The impact of using reconditioned correlated observation‐error covariance matrices in the Met Office 1D‐Var system

Cited by 13 publications

References 44 publications

New bounds on the condition number of the Hessian of the preconditioned variational data assimilation problem

New bounds on the condition number of the Hessian of the preconditioned variational data assimilation problem

The conditioning of least‐squares problems in variational data assimilation

Estimation of the error covariance matrix for IASI radiances and its impact on ozone analyses

Contact Info

Product

Resources

About