Representation, representativity, representativeness error, forward interpolation error, forward model error, observation-operator error, aggregation error and sampling error are all terms used to refer to components of observation error in the context of data assimilation. This article is an attempt to consolidate the terminology that has been used in the earth sciences literature and was suggested at a European Space Agency workshop held in Reading in April 2014. We review the state of the art and, through examples, motivate the terminology. In addition to a theoretical framework, examples from application areas of satellite data assimilation, ocean reanalysis and atmospheric chemistry data assimilation are provided. Diagnosing representation-error statistics as well as their use in state-of-the-art data assimilation systems is discussed within a consistent framework.

SummaryIn variational data assimilation a least-squares objective function is minimised to obtain the most likely state of a dynamical system. This objective function combines observation and prior (or background) data weighted by their respective error statistics. In numerical weather prediction, data assimilation is used to estimate the current atmospheric state, which then serves as an initial condition for a forecast. New developments in the treatment of observation uncertainties have recently been shown to cause convergence problems for this least-squares minimisation. This is important for operational numerical weather prediction centres due to the time constraints of producing regular forecasts. The condition number of the Hessian of the objective function can be used as a proxy to investigate the speed of convergence of the least-squares minimisation. In this paper we develop novel theoretical bounds on the condition number of the Hessian. These new bounds depend on the minimum eigenvalue of the observation error covariance matrix and the ratio of background error variance to observation error variance. Numerical tests in a linear setting show that the location of observation measurements has an important effect on the condition number of the Hessian.We identify that the conditioning of the problem is related to the complex interactions between observation error covariance and background error covariance matrices. Increased understanding of the role of each constituent matrix in the conditioning of the Hessian will prove useful for informing the choice of correlated observation error covariance matrix and observation location, particularly for practical applications.

With the development of convection-permitting numerical weather prediction the efficient use of high-resolution observations in data assimilation is becoming increasingly important. The operational assimilation of these observations, such as Doppler radar radial winds (DRWs), is now common, although to avoid violating the assumption of uncorrelated observation errors the observation density is severely reduced. To improve the quantity of observations used and the impact that they have on the forecast requires the introduction of the full, potentially correlated, error statistics. In this work, observation error statistics are calculated for the DRWs that are assimilated into the Met Office high-resolution U.K. model (UKV) using a diagnostic that makes use of statistical averages of observation-minus-background and observation-minus-analysis residuals. This is the first in-depth study using the diagnostic to estimate both horizontal and along-beam observation error statistics. The new results obtained show that the DRW error standard deviations are similar to those used operationally and increase as the observation height increases. Surprisingly, the estimated observation error correlation length scales are longer than the operational thinning distance. They are dependent both on the height of the observation and on the distance of the observation away from the radar. Further tests show that the long correlations cannot be attributed to the background error covariance matrix used in the assimilation, although they are, in part, a result of using superobservations and a simplified observation operator. The inclusion of correlated error statistics in the assimilation allows less thinning of the data and hence better use of the high-resolution observations.

It has been common practice in data assimilation to treat observation errors as uncorrelated; however, meteorological centres are beginning to use correlated inter-channel observation errors in their operational assimilation systems. In this work, we are the first to characterise inter-channel and spatial error correlations for Spinning Enhanced Visible and Infrared Imager (SEVIRI) observations that are assimilated into the Met Office high-resolution model. The errors are calculated using a diagnostic that calculates statistical averages of observation-minus-background and observation-minus-analysis residuals. This diagnostic is sensitive to the background and observation error statistics used in the assimilation, although, with careful interpretation of the results, it can still provide useful information. We find that the diagnosed SEVIRI error variances are as low as one-tenth of those currently used in the operational system. The water vapour channels have significantly correlated inter-channel errors, as do the surface channels. The surface channels have larger observation error variances and inter-channel correlations in coastal areas of the domain; this is the result of assimilating mixed pixel (land-sea) observations. The horizontal observation error correlations range between 30 km and 80 km, which is larger than the operational thinning distance of 24 km. We also find that estimates from the diagnostics are unaffected by biased observations, provided that the observation-minus-background and observation-minus-analysis residual means are subtracted.

To improve the quantity and impact of observations used in data assimilation, it is necessary to take into account the full, potentially correlated, observation error statistics. A number of methods for estimating correlated observation errors exist, but a popular method is a diagnostic that makes use of statistical averages of observation-minus-background and observation-minus-analysis residuals. The accuracy of the results it yields is unknown as the diagnostic is sensitive to the difference between the exact background and exact observation error covariances and those that are chosen for use within the assimilation. It has often been stated in the literature that the results using this diagnostic are only valid when the background and observation error correlation length-scales are well separated. Here we develop new theory relating to the diagnostic. For observations on a 1D periodic domain we are able to the show the effect of changes in the assumed error statistics used in the assimilation on the estimated observation error covariance matrix. We also provide bounds for the estimated observation error variance and eigenvalues of the estimated observation error correlation matrix. We demonstrate that it is still possible to obtain useful results from the diagnostic when the background and observation error length-scales are similar. In general, our results suggest that when correlated observation errors are treated as uncorrelated in the assimilation, the diagnostic will underestimate the correlation lengthscale. We support our theoretical results with simple illustrative examples. These results have potential use for interpreting the derived covariances estimated using an operational system.

This article is published with the permission of the Controller of HMSO and the Queen's Printer for Scotland.The observation-error covariance matrix used in data assimilation contains contributions from instrument errors, representativity errors and errors introduced by the approximated observation operator. Forward model errors arise when the observation operator does not correctly model the observations or when observations can resolve spatial scales that the model cannot. Previous work to estimate the observation-error covariance matrix for particular observing instruments has shown that it contains significant correlations. In particular, correlations for humidity data are more significant than those for temperature. However it is not known what proportion of these correlations can be attributed to the representativity errors. In this article we apply an existing method for calculating representativity error, previously applied to an idealised system, to NWP data. We calculate horizontal errors of representativity for temperature and humidity using data from the Met Office high-resolution UK variable resolution model. Our results show that errors of representativity are correlated and more significant for specific humidity than temperature. We also find that representativity error varies with height. This suggests that the assimilation scheme may be improved if these errors are explicitly included in a data assimilation scheme.

The importance of prior error correlations in data assimilation has long been known; however, observation-error correlations have typically been neglected. Recent progress has been made in estimating and accounting for observation-error correlations, allowing for the optimal use of denser observations. Given this progress, it is now timely to ask how prior and observation-error correlations interact and how this affects the value of the observations in the analysis. Addressing this question is essential to understanding the optimal design of future observation networks for high-resolution numerical weather prediction. This article presents new results, which unify and advance upon previous studies on this topic.The interaction of the prior and observation-error correlations is illustrated with a series of two-variable experiments in which the mapping between the state and observed variables (the observation operator) is allowed to vary. In an optimal system, the reduction in the analysis-error variance and spread of information is shown to be greatest when the observation and prior errors have complementary statistics: for example, in the case of direct observations, when the correlations between the observation and prior errors have opposite signs. This can be explained in terms of the relative uncertainty of the observations and prior on different spatial scales. The results from these simple two-variable experiments are used to inform the optimal observation density for observations of a circular domain (with 32 grid points). It is found that dense observations are most beneficial when they provide a more accurate estimate of the state at smaller scales than the prior estimate. In the case of second-order auto-regressive correlation functions, this is achieved when the length-scales of the observation-error correlations are greater than those of the prior estimate and the observations are direct measurements of the state variables.

No abstract