A pragmatic strategy for implementing spatially correlated observation errors in an operational system: An application to Doppler radial winds

Simonin, David; Waller, John C.; Ballard, Su; Dance, Sarah L.; Nichols, Nancy K.

doi:10.1002/qj.3592

Cited by 29 publications

(59 citation statements)

References 63 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Depending on the application, a variety of methods have been used to combat the problem of rank deficiency of sample covariance matrices. In the case of spatially correlated errors it may be possible to fit a smooth correlation function or operator to the sample covariance matrix as was done in Simonin et al [2019], Guillet et al [2019] respectively. Another approach is to retain only the first k leading eigenvectors of the estimated correlation matrix and to add a diagonal matrix to ensure the resulting covariance matrix has full rank [Michel, 2018, Stewart et al, 2013.…”

Section: Introductionmentioning

confidence: 99%

Improving the condition number of estimated covariance matrices

Tabeart¹,

Dance²,

Lawless³

et al. 2020

Tellus A: Dynamic Meteorology and Oceanography

Self Cite

View full text Add to dashboard Cite

High dimensional error covariance matrices and their inverses are used to weight the contribution of observation and background information in data assimilation procedures. As observation error covariance matrices are often obtained by sampling methods, estimates are often degenerate or ill-conditioned, making it impossible to invert an observation error covariance matrix without the use of techniques to reduce its condition number. In this paper we present new theory for two existing methods that can be used to 'recondition' any covariance matrix: ridge regression, and the minimum eigenvalue method. We compare these methods with multiplicative variance inflation, which cannot alter the condition number of a matrix, but is often used to account for neglected correlation information. We investigate the impact of reconditioning on variances and correlations of a general covariance matrix in both a theoretical and practical setting. Improved theoretical understanding provides guidance to users regarding method selection, and choice of target condition number. The new theory shows that, for the same target condition number, both methods increase variances compared to the original matrix, with larger increases for ridge regression than the minimum eigenvalue method. We prove that the ridge regression method strictly decreases the absolute value of off-diagonal correlations. Theoretical comparison of the impact of reconditioning and multiplicative variance inflation on the data assimilation objective function shows that variance inflation alters information across all scales uniformly, whereas reconditioning has a larger effect on scales corresponding to smaller eigenvalues. We then consider two examples: a general correlation function, and an observation error covariance matrix arising from interchannel correlations. The minimum eigenvalue method results in smaller overall changes to the correlation matrix than ridge regression, but can increase off-diagonal correlations. Data assimilation experiments reveal that reconditioning corrects spurious noise in the analysis but underestimates the true signal compared to multiplicative variance inflation.

show abstract

Section: Introductionmentioning

confidence: 99%

Improving the condition number of estimated covariance matrices

Tabeart¹,

Dance²,

Lawless³

et al. 2020

Tellus A: Dynamic Meteorology and Oceanography

Self Cite

View full text Add to dashboard Cite

show abstract

“…One such set of observations are Doppler radar radial winds (DRWs). The assimilation of DRWs provides a significant positive impact on the forecast (Montmerle and Faccani 2009;Xue et al 2013Xue et al , 2014 and as a result they are now assimilated at a number of operational centers (Xiao et al 2008;Simonin et al 2014). However, to effectively assimilate high-resolution observations it is necessary to understand and correctly account for their error statistics (Gorin and Tsyrulnikov 2011).…”

Section: Introductionmentioning

confidence: 99%

Observation Error Statistics for Doppler Radar Radial Wind Superobservations Assimilated into the DWD COSMO-KENDA System

Waller

Bauernschubert

Dance

et al. 2019

Monthly Weather Review

View full text Add to dashboard Cite

Currently in operational numerical weather prediction (NWP) the density of high-resolution observations, such as Doppler radar radial winds (DRWs), is severely reduced in part to avoid violating the assumption of uncorrelated observation errors. To improve the quantity of observations used and the impact that they have on the forecast requires an accurate specification of the observation uncertainties. Observation uncertainties can be estimated using a simple diagnostic that utilizes the statistical averages of observation-minus-background and observation-minus-analysis residuals. We are the first to use a modified form of the diagnostic to estimate spatial correlations for observations used in an operational ensemble data assimilation system. The uncertainties for DRW superobservations assimilated into the Deutscher Wetterdienst convection-permitting NWP model are estimated and compared to previous uncertainty estimates for DRWs. The new results show that most diagnosed standard deviations are smaller than those used in the assimilation, hence, it may be feasible to assimilate DRWs using reduced error standard deviations. However, some of the estimated standard deviations are considerably larger than those used in the assimilation; these large errors highlight areas where the observation processing system may be improved. The error correlation length scales are larger than the observation separation distance and influenced by both the superobbing procedure and observation operator. This is supported by comparing these results to our previous study using Met Office data. Our results suggest that DRW error correlations may be reduced by improving the superobbing procedure and observation operator; however, any remaining correlations should be accounted for in the assimilation.

show abstract

“…Still, Figure 6a demonstrates that the analysis can further be improved by assimilating most or all the available observations, if the variances are optimally inflated. As in Fowler et al (2018) and Simonin et al (2019), the analysis error STD is reduced when increasing the density of assimilated F I G U R E 7 Same as Figure 5, but for the case assimilating spatial difference observations computed from observations with spatially correlated errors with a full (considering spatial correlations) and a diagonal (neglecting spatial correlations) observation error covariance matrix observations by retrieving some of the small-scale information that is lost when thinning the observations.…”

Section: Direct Approachmentioning

confidence: 93%

“…Spatially correlated observation errors can be explicitly accounted for by computing and using its Cholesky decomposition (e.g. Simonin et al ., ). Also, Guillet et al .…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A practical assimilation approach to extract smaller‐scale information from observations with spatially correlated errors: An idealized study

Bédard

Buehner

2019

Quart J Royal Meteoro Soc

View full text Add to dashboard Cite

It is still common to neglect the spatial error correlations of assimilated observations in numerical weather prediction systems because no practical approach is available to account for them when the number of observations with correlated error is large or when these observations are non‐uniformly distributed. Instead, it is common practice to inflate observation error variances to avoid overfitting large scales and spatially thin observations to reduce error correlations between remaining observations, although both methods generally sacrifice small‐scale information. Inspired by previous work on assimilating the difference between adjacent observations (so‐called spatial difference observations), this study aims at combining direct observations with spatial difference observations in the assimilation to extract both large‐ and smaller‐scale information from observations with spatially correlated errors, while still neglecting these error correlations. Experiments performed in a simplified 1D context over a periodic domain show that the combined approach is numerically equivalent to directly assimilating observations available at every grid point using non‐diagonal observation error covariances based on a first‐order autoregressive correlation function. In a case where observation error correlations have a different structure (e.g. Gaussian), the true observation error correlations are not perfectly taken into account by the combined approach, but it still efficiently extracts information to correct the scales that have the most errors. Combining direct observations with spatial difference observations proves complementary and experimental results show lower analysis errors for both large and intermediate scales. More specifically, while neglecting spatial observation error correlations, the combined approach provides results with lower analysis error than the direct approach, especially when spatial error correlations are large and when direct observations are spatially thinned.

show abstract

A pragmatic strategy for implementing spatially correlated observation errors in an operational system: An application to Doppler radial winds

Cited by 29 publications

References 63 publications

Improving the condition number of estimated covariance matrices

Improving the condition number of estimated covariance matrices

Observation Error Statistics for Doppler Radar Radial Wind Superobservations Assimilated into the DWD COSMO-KENDA System

A practical assimilation approach to extract smaller‐scale information from observations with spatially correlated errors: An idealized study

Contact Info

Product

Resources

About