A mechanism for catastrophic filter divergence in data assimilation for sparse observation networks

Gottwald, Georg A.; Majda, Andrew J.

doi:10.5194/npg-20-705-2013

Cited by 39 publications

(59 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In particular, even when the true model signal is stationary at the origin, the filter estimate diverges exponentially fast to infinity in a phenomena known as catastrophic filter divergence (7,8). The mechanism that creates this filter malfunction is very intuitive and is illustrated by a simple picture.…”

Section: Discussionmentioning

confidence: 99%

“…Despite their widespread application, the theoretical understanding of these methods remains underdeveloped. Recent efforts have been made to understand the dynamical properties of EnKF/ESRF in the practical setting of high-dimensional turbulent forecast models with low ensemble size, focusing on well-posedness (6) and stability.One of the main motivations for these theoretical studies was the curious numerical phenomenon known as catastrophic filter divergence (7,8). In refs.…”

mentioning

confidence: 99%

“…One of the main motivations for these theoretical studies was the curious numerical phenomenon known as catastrophic filter divergence (7,8). In refs.…”

mentioning

confidence: 99%

“…The mechanism behind the blowup validates the intuition from ref. 8 and involves an interplay between model stiffness and ensemble alignment.…”

mentioning

confidence: 99%

See 3 more Smart Citations

Concrete ensemble Kalman filters with rigorous catastrophic filter divergence

Kelly

Majda

Tong

2015

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

The ensemble Kalman filter and ensemble square root filters are data assimilation methods used to combine high-dimensional, nonlinear dynamical models with observed data. Ensemble methods are indispensable tools in science and engineering and have enjoyed great success in geophysical sciences, because they allow for computationally cheap low-ensemble-state approximation for extremely high-dimensional turbulent forecast models. From a theoretical perspective, the dynamical properties of these methods are poorly understood. One of the central mysteries is the numerical phenomenon known as catastrophic filter divergence, whereby ensemble-state estimates explode to machine infinity, despite the true state remaining in a bounded region. In this article we provide a breakthrough insight into the phenomenon, by introducing a simple and natural forecast model that transparently exhibits catastrophic filter divergence under all ensemble methods and a large set of initializations. For this model, catastrophic filter divergence is not an artifact of numerical instability, but rather a true dynamical property of the filter. The divergence is not only validated numerically but also proven rigorously. The model cleanly illustrates mechanisms that give rise to catastrophic divergence and confirms intuitive accounts of the phenomena given in past literature. data assimilation | ensemble Kalman filter | filter divergence W ith the growing importance of accurate weather forecasting and the expanding availability of geophysical measurements, data assimilation has never been more vital to society. Ensemblebased assimilation methods, including the ensemble Kalman filter (EnKF) (1) and ensemble square root filters (ESRF) (2, 3), are crucial components of data assimilation that are applied ubiquitously across the geophysical sciences (4, 5). Despite their widespread application, the theoretical understanding of these methods remains underdeveloped. Recent efforts have been made to understand the dynamical properties of EnKF/ESRF in the practical setting of high-dimensional turbulent forecast models with low ensemble size, focusing on well-posedness (6) and stability.One of the main motivations for these theoretical studies was the curious numerical phenomenon known as catastrophic filter divergence (7,8). In refs. 7 and 8, it was numerically demonstrated that state estimates provided by ensemble-based methods can explode to machine infinity, despite the forecast model's being dissipative and satisfying the absorbing ball property (9), which demands that the true state be always absorbed by a bounded region of the state space. The genesis of this phenomenon has previously been attributed to an interplay between stiffness in the forecast models and forecast ensemble alignment. Nevertheless, until now there has been no concrete example that transparently illustrates the mechanism behind catastrophic filter divergence. Without an explicit example or concrete understanding of the phenomenon, it is difficult to identify which models are vul...

show abstract

Section: Discussionmentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

Concrete ensemble Kalman filters with rigorous catastrophic filter divergence

Kelly

Majda

Tong

2015

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

show abstract

“…Gottwald and Majda (2013) explained this instability by the contribution of the unrealistic large covariance terms. In order to damp the rapid temporal change of model parameter, we set,…”

Section: Parameter Evolution and Uncertaintiesmentioning

confidence: 99%

High-resolution data assimilation through stochastic subgrid tensor and parameter estimation from 4DEnVar

Yin¹,

Mémin

2017

Tellus A: Dynamic Meteorology and Oceanography

View full text Add to dashboard Cite

In this paper, we explore a dynamical formulation allowing to assimilate high-resolution data in a large-scale fluid flow model. This large-scale formulation relies on a random modelling of the small-scale velocity component and allows to take into account the scale discrepancy between the dynamics and the observations. It introduces a subgrid stress tensor that naturally emerges from a modified Reynolds transport theorem adapted to this stochastic representation of the flow. This principle is used within a stochastic shallow water model coupled with an 4DEnVar assimilation technique to estimate both the flow initial conditions and the inhomogeneous time-varying subgrid parameters. The performance of this modelling has been assessed numerically with both synthetic and real-world data. Our strategy has shown to be very effective in providing a more relevant prior/posterior ensemble in terms of the dispersion compared to other tests using the standard shallow water equations with no subgrid parameterization or with simple eddy viscosity models. We also compared two localization techniques. The results indicate the localized covariance approach is more suitable to deal with the scale discrepancy-related errors.

show abstract