Why does EnKF suffer from analysis overconfidence? An insight into exploiting the ever‐increasing volume of observations

Abstract

Ensemble Kalman filters (EnKF) are empirically known to suffer from insufficient posterior spread and this issue is aggravated when assimilating a large volume of observations. This problem, commonly called analysis underdispersion or analysis overconfidence, has been well recognized, but why it occurs seems to be rather poorly understood. Inspired by the theory of the degrees of freedom for signal, this article investigates this problem by analyzing the trace of the matrix HK, where H and K represent, respectively, the observation operator and the gain matrix. A simple mathematical argument shows that tr HK for EnKF is bounded from above by the ensemble size, which entails that assimilating many more observations than the ensemble size leads automatically to tr HK underestimation, as long as the observations are of accuracy comparable to the background. Since tr HK can be interpreted as the squared spread of the posterior ensemble measured in the normalized observation space, underestimated tr HK implies overconfidence in the analysis spread, which, in a cycled context, requires covariance inflation to be applied. The theory is then extended to cases where covariance localization schemes (either B‐localization or R‐localization) are applied to show how they alleviate the analysis underdispersion. These findings from the mathematical argument are demonstrated with a simple one‐dimensional covariance model. Finally, the findings described above are used to form speculative arguments about how to interpret several puzzling features of the local ensemble transform Kalman filter (LETKF) previously reported in the literature, such as why using fewer observations can lead to better performance, when optimal localization scales tend to occur, and why covariance inflation methods based on the relaxation to prior information approach are particularly successful when observations are distributed inhomogeneously.

Implementing Hybrid Background Error Covariance into the LETKF with Attenuation-Based Localization: Experiments with a Simplified AGCM

“…available (Hamrud et al 2015;Shraff et al 2016). This fact is partly caused by Rlocalization, which is the most commonly used localization approach for the LETKF (Hotta and Ota 2021). This issue may be overcome through the model space localization by ensemble modulation approach Hodyss 2009a, 2009b)…”

Advantage of 30‐s‐Updating Numerical Weather Prediction With a Phased‐Array Weather Radar Over Operational Nowcast for a Convective Precipitation System

“…For the innermost domain, radar reflectivity and Doppler velocity observations from the MP‐PAWR have been assimilated with the relaxation to prior spread method (Whitaker & Hamill, 2012) with a coefficient of 0.95. Recent studies indicated that the ensemble size limits the number of observations that can be effectively assimilated by the ensemble Kalman filter (Hamrud et al., 2015; Hotta & Ota, 2021; Schraff et al., 2016), so that this study set the observation number limit for each grid at a total of 200 (100 for radar reflectivity and 100 for Doppler velocity). The standard deviation of the localization scales (Equation 13 in Miyoshi and Yamane (2007)) is set to be 2 km both horizontally and vertically, corresponding to ∼7.3 km of the cutoff radius where analysis increments become zero.…”

Development of the Real‐Time 30‐s‐Update Big Data Assimilation System for Convective Rainfall Prediction With a Phased Array Weather Radar: Description and Preliminary Evaluation

“…Recently, several studies indicated that the ensemble size limits the number of observations that can be effectively assimilated at each grid by the ensemble Kalman filter (EnKF; Hamrud et al., 2015; Hotta & Ota, 2021; Schraff et al., 2016). For example, Schraff et al.…”

“…Superobbing generally reduces the observation error, but its correlated component remains (Janjić et al., 2018). In addition, previous studies indicated that the ensemble size limits the number of observations that can be effectively assimilated at each grid by the EnKF (Hamrud et al., 2015; Hotta & Ota, 2021; Schraff et al., 2016). Therefore, to reduce the effects of the remained observation error correlations in the superobbed MP‐PAWR data with the observation number limit, we apply a new thinning method to the superobbed MP‐PAWR observations.…”

“…Recently, several studies indicated that the ensemble size limits the number of observations that can be effectively assimilated at each grid by the ensemble Kalman filter (EnKF; Hamrud et al, 2015;Hotta & Ota, 2021;Schraff et al, 2016). For example, Schraff et al (2016) used an adaptive localization method that keeps the effective number of assimilated observations at each grid at a constant.…”