A dynamical Gaussian, lognormal, and reverse lognormal Kalman filter

Abstract

We derive a generalization of the Kalman filter that allows for non‐Gaussian background and observation errors. The Gaussian assumption is replaced by considering that the errors come from a mixed distribution of Gaussian, lognormal, and reverse lognormal random variables. We detail the derivation for reverse lognormal errors, and extend the results to mixed distributions, where the number of Gaussian, lognormal, and reverse lognormal state variables can dynamically change every analysis time. We robustly test the dynamical mixed Kalman filter on two different systems based on the Lorenz 1963 model, and demonstrate that non‐Gaussian techniques generally improve the analysis skill if the observations are sparse and uncertain, compared to the Gaussian Kalman filter.This article is protected by copyright. All rights reserved.

Foundations for Universal Non‐Gaussian Data Assimilation

“…The variational method proposed in this work is a first step toward non‐Gaussian hybrid approaches. It has been shown that Kalman filters (Fletcher et al., 2023; Van Loon & Fletcher, 2023) and the maximum likelihood ensemble filter (Zupanski, 2005) can be adapted to (reverse) lognormal errors. As these distributions are part of the Johnson family, our generalized framework can be applied to such techniques, which is the subject of future work.…”

“…Note that the semi‐bounded Johnson distribution is equivalent to the lognormal distribution when λ > 0 and to the reverse lognormal distribution when λ < 0. These distributions have already been successfully used in different data assimilation settings, such as variational techniques (Fletcher & Zupanski, 2006, 2007; Kliewer et al., 2016), Kalman filters (Fletcher et al., 2023; Van Loon & Fletcher, 2023), and ensemble methods. A major advantage of the bounded and unbounded distributions is that their skewness can change from positive to negative by changing μ , see Figures 1a and 1c.…”

Dynamical Gaussian, lognormal, and reverse lognormal maximum‐likelihood ensemble filter

“…In particular, we show that correctly assuming a (reverse) lognormal distribution for the assimilation of nonlinear observations in the Lorenz-05 model II (Lorenz, 2005) leads to lower mean absolute errors in the analysis state, when compared with a known truth. Moreover, we test the full power of the dynamical Gaussian, lognormal, and reverse lognormal MLEF in a coupled version of the Lorenz-63 model (Lorenz, 1963;Van Loon & Fletcher, 2024), and demonstrate an improved analysis skill when relaxing the Gaussian assumption.…”

“…To solve the former, Goodliff et al (2022) proposed to use machine learning to be able to switch between distributions, and demonstrated that this improved the analysis skill in the Lorenz-63 model. A crucial step in solving the latter was made by Fletcher et al (2023), where a mixed Gaussian-lognormal version of the Kalman filter was derived and tested; this was later extended to include reverse lognormal variables (Van Loon & Fletcher, 2024). Here, we expand further on these ideas to develop a non-Gaussian version of the maximum-likelihood ensemble filter (Zupanski, 2005), advancing towards operationally viable non-Gaussian data assimilation.…”

“…Whereas in 3DVAR the cost function is used directly to find the optimal estimate, in the MLEF a clever transformation is performed that introduces ensemble members. Using the Kalman filter equations (Fletcher et al, 2023;Jazwinski, 1970;Van Loon & Fletcher, 2024), the forecast-error covariance matrix P f can be calculated by evolving the analysis-error covariance matrix P a from the previous analysis step in the linearised model M and written in a square root form as…”