OpenDA-NEMO framework for ocean data assimilation

Velzen, Nils van; Altaf, Muhammad Umer; Verlaan, Martin

doi:10.1007/s10236-016-0945-z

Cited by 19 publications

(8 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Frameworks relying on the off-line mode include EnTK [39], with the largest published DA use cases reaching 4,096 members for a molecular dynamics application with an EnKF filter [40]. OpenDA also follows this model, using NetCDF for data exchange with the NEMO code [41]. DART supports both [42], with reports of large scale DA in off-line mode in [43] (about 1,000 members with an oceanic code), or [44,45] (1,024 member, LETKF filter, 6 M Fugaku cores).…”

Section: Related Workmentioning

confidence: 99%

A Framework for Large Scale Particle Filters Validated with Data Assimilation for Weather Simulation

Friedemann,

Keller,

et al. 2023

Preprint

View full text Add to dashboard Cite

Particle filters are a group of algorithms to solve inverse problems through statistical Bayesian methods when the model does not comply with the linear and Gaussian hypothesis. Particle filters are used in domains like data assimilation, probabilistic programming, neural network optimization, localization and navigation. Particle filters estimate the probability distribution of model states by running a large number of model instances, the so called particles. The ability to handle a very large number of particles is critical for high dimensional models. This paper proposes a novel paradigm to run very large ensembles of parallel model instances on supercomputers. The approach combines an elastic and fault tolerant runner/server model minimizing data movements while enabling dynamic load balancing. Particle weights are computed locally on each runner and transmitted when available to a server that normalizes them, resamples new particles based on their weight, and redistributes dynamically the work to runners to react to load imbalance. Our approach relies on a an asynchronously managed distributed particle cache permitting particles to move from one runner to another in the background while particle propagation goes on. This also enables the number of runners to vary during the execution either in reaction to failures and

show abstract

Section: Related Workmentioning

confidence: 99%

A Framework for Large Scale Particle Filters Validated with Data Assimilation for Weather Simulation

Friedemann,

Keller,

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…A file-based approach may be the only option available when the full-supercomputer is required to propagate just a fraction of members as in Yashiro et al (2020). The OpenDA framework also supports this file-based model, relying on NetCDF files for data exchange for the NEMO ocean model in Van Velzen et al (2016). However, relying on the machine I/O capabilities using files is a growing performance bottleneck.…”

Section: Related Workmentioning

confidence: 99%

An elastic framework for ensemble-based large-scale data assimilation

Friedemann

Raffin

2022

The International Journal of High Performance Computing Applica

View full text Add to dashboard Cite

Prediction of chaotic systems relies on a floating fusion of sensor data (observations) with a numerical model to decide on a good system trajectory and to compensate non-linear feedback effects. Ensemble-based data assimilation (DA) is a major method for this concern depending on propagating an ensemble of perturbed model realizations. In this paper, we develop an elastic, online, fault-tolerant and modular framework called Melissa-DA for large-scale ensemble-based DA. Melissa-DAallows elastic addition or removal of compute resources for state propagation at runtime. Dynamic load balancing based on list scheduling ensures efficient execution. Online processing of the data produced by ensemble members enables to avoid the I/O bottleneck of file-based approaches. Our implementation embeds the PDAF parallel DA engine, enabling the use of various DA methods. Melissa-DAcan support extra ensemble-based DA methods by implementing the transformation of member background states into analysis states. Experiments confirm the excellent scalability of Melissa-DA, propagating 16,384 members for a regional hydrological critical zone assimilation relying on the ParFlow model on a domain with about 4 M grid cells. The same use case was ported to the PDAF state-of-the-art DA framework relying on a MPI approach. A comparison with Melissa-DA at 2500 members on 20,000 cores shows our approach is about 50% faster per assimilation cycle.

show abstract

“…As a comparison, a number of frameworks exist to integrate DA with existing models. Examples include the Data Assimilation Research Testbed from the National Center for Atmospheric Research (NCAR, Anderson et al, 2009), the Javabased OpenDA (van Velzen et al, 2016), and the Parallel Data Assimilation Framework (Nerger et al, 2005) developed by the Alfred Wegener Institute.…”

Section: Motivationmentioning

confidence: 99%

An integrated framework that combines machine learning and numerical models to improve wave-condition forecasts

O’Donncha

Zhang

Chen

et al. 2018

Journal of Marine Systems

View full text Add to dashboard Cite

This study investigates near-shore circulation and wave characteristics applied to a case-study site in Monterey Bay, California. We integrate physicsbased models to resolve wave conditions together with a machine-learning algorithm that combines forecasts from multiple, independent models into a single "best-estimate" prediction of the true state. The Simulating WAves Nearshore (SWAN) physics-based model is used to compute wind-augmented waves. Ensembles are developed based on multiple simulations perturbing data input to the model. A learning-aggregation technique uses historical observations and model forecasts to calculate a weight for each ensemble member. We compare the weighted ensemble predictions with measured data to evaluate performance against present state-of-the-art. Finally, we discuss how this framework that integrates data-driven and physics-based approaches can outperform either technique in isolation.

show abstract

OpenDA-NEMO framework for ocean data assimilation

Cited by 19 publications

References 31 publications

A Framework for Large Scale Particle Filters Validated with Data Assimilation for Weather Simulation

A Framework for Large Scale Particle Filters Validated with Data Assimilation for Weather Simulation

An elastic framework for ensemble-based large-scale data assimilation

An integrated framework that combines machine learning and numerical models to improve wave-condition forecasts

Contact Info

Product

Resources

About