Latent space data assimilation by using deep learning

Abstract

Performing data assimilation (DA) at low cost is of prime concern in Earth system modeling, particularly in the era of Big Data, where huge quantities of observations are available. Capitalizing on the ability of neural network techniques to approximate the solution of partial differential equations (PDEs), we incorporate deep learning (DL) methods into a DA framework. More precisely, we exploit the latent structure provided by autoencoders (AEs) to design an ensemble transform Kalman filter with model error (ETKF‐Q) in the latent space. Model dynamics are also propagated within the latent space via a surrogate neural network. This novel ETKF‐Q‐Latent (ETKF‐Q‐L) algorithm is tested on a tailored instructional version of Lorenz 96 equations, named the augmented Lorenz 96 system, which possesses a latent structure that accurately represents the observed dynamics. Numerical experiments based on this particular system evidence that the ETKF‐Q‐L approach both reduces the computational cost and provides better accuracy than state‐of‐the‐art algorithms such as the ETKF‐Q.

Equation‐Free Surrogate Modeling of Geophysical Flows at the Intersection of Machine Learning and Data Assimilation

“…However, in this study, the forward model is replaced with the surrogate model, and therefore, we perform the DA in the latent space. The similar ideas have also been used in other studies that deals with the DA for reduced-order models (Amendola et al, 2020;Maulik et al, 2022;Peyron et al, 2021). Once the LSTM network is trained, it is used to forecast the future state of the POD modal coefficients in an auto-regressive manner .…”

Ensemble latent assimilation with deep learning surrogate model: application to drop interaction in a microfluidics device

“…Once the prediction using surrogate ML models has been performed, latent data assimilation (also known as latent assimilation (LA)) 30,53 is employed to adjust the model output on the latent space via real-time observations. DA is widely employed for analyzing model predictions, (also known as background states or a priori estimates) and observations to determine the most probable state ( a posteriori estimate).…”

“…assigning B̃ t and R̃ t equal to identity matrix I N LS × N LS . 30,53 In this study, performing DA with identity matrices is set as the benchmark, and more sophisticated methods ( e.g. NMC, ensemble) of estimating B̃ t are employed.…”

A Tutorial on Bayesian Data Assimilation

“…This section follows this latest paper, showing how to derive the cost function of the generalised estimation problem. Note that going even beyond this approach, attempting to learn the optimisation scheme of the cost function, or even the full DA procedure, an approach called end-to-end in ML, is a subject of active investigations [Fablet et al, 2021, Peyron et al, 2021.…”