Improving the condition number of estimated covariance matrices

Abstract

High dimensional error covariance matrices and their inverses are used to weight the contribution of observation and background information in data assimilation procedures. As observation error covariance matrices are often obtained by sampling methods, estimates are often degenerate or ill-conditioned, making it impossible to invert an observation error covariance matrix without the use of techniques to reduce its condition number. In this paper we present new theory for two existing methods that can be used to 'recondition' any covariance matrix: ridge regression, and the minimum eigenvalue method. We compare these methods with multiplicative variance inflation, which cannot alter the condition number of a matrix, but is often used to account for neglected correlation information. We investigate the impact of reconditioning on variances and correlations of a general covariance matrix in both a theoretical and practical setting. Improved theoretical understanding provides guidance to users regarding method selection, and choice of target condition number. The new theory shows that, for the same target condition number, both methods increase variances compared to the original matrix, with larger increases for ridge regression than the minimum eigenvalue method. We prove that the ridge regression method strictly decreases the absolute value of off-diagonal correlations. Theoretical comparison of the impact of reconditioning and multiplicative variance inflation on the data assimilation objective function shows that variance inflation alters information across all scales uniformly, whereas reconditioning has a larger effect on scales corresponding to smaller eigenvalues. We then consider two examples: a general correlation function, and an observation error covariance matrix arising from interchannel correlations. The minimum eigenvalue method results in smaller overall changes to the correlation matrix than ridge regression, but can increase off-diagonal correlations. Data assimilation experiments reveal that reconditioning corrects spurious noise in the analysis but underestimates the true signal compared to multiplicative variance inflation.

The impact of using reconditioned correlated observation‐error covariance matrices in the Met Office 1D‐Var system

“…However, the specific OEC matrix being used was extremely ill‐conditioned. We expect that reconditioning will mitigate the impact on computation time, as well as the number of iterations required for convergence of the 1D‐Var assimilation, in a similar manner to the improvements seen for 4D‐Var (Weston, ; Tabeart et al ., ).…”

“…Increasing the amount of reconditioning applied to correlated OEC matrices improves convergence of the 1D‐Var routine, in accordance with the qualitative theoretical conclusions of Tabeart et al . (; ). Most experimental choices of correlated OEC matrix resulted in a larger number of IASI observations that were accepted by the 1D‐Var routine than the current diagonal operational choice. Increasing the amount of reconditioning applied to correlated OEC matrices increases the number of IASI observations that converge in fewer than 10 iterations, and hence pass the quality‐control component of 1D‐Var. Retrieval differences for skin temperature, cloud fraction, and cloud‐top pressure are smaller than retrieved standard deviation values for over 75% of IASI observations for all choices of correlated OEC matrix.…”

“…Tabeart et al . () also studied the impact of using the ridge regression method of reconditioning on the variational objective function (Equation ). Use of the ridge regression method was found to reduce the weight on all observation information in the analysis, with a relatively larger effect on information associated with small eigenvalues.…”

“…These methods were investigated theoretically in Tabeart et al . (), where it was found that both methods increase standard deviations, and that the ridge regression method reduces all off‐diagonal correlations strictly. Both methods were compared in an operational system in Campbell et al .…”

Hierarchical Canonical Correlation Analysis Reveals Phenotype, Genotype, and Geoclimate Associations in Plants

“…Thus, in the application of the HCCA algorithm, we propose to also estimate the hyperparameters from cross-covariance analysis by the condition number. When HCCA runs CCA without PPI network as in equation (1) withC ii = C ii + αI, to makeC ii well-conditioned, we choose α such thatC ii has a desirable condition number using the technique of reconditioning [20]. Specifically, given c as the desirable condition number ofC ii , α can be chosen as…”