Large scale in transit computation of quantiles for ensemble runs

Abstract

The classical approach for quantiles computation requires availability of the full sample before ranking it. In uncertainty quantification of numerical simulation models, this approach is not suitable at exascale as large ensembles of simulation runs would need to gather a prohibitively large amount of data. This problem is solved thanks to an on-the-fly and iterative approach based on the Robbins-Monro algorithm. This approach relies on Melissa, a file avoiding, adaptive, fault-tolerant and elastic framework. On a validation case producing 11 TB of data, which consists in 3000 fluid dynamics parallel simulations on a 6M cell mesh, it allows on-line computation of spatio-temporal maps of percentiles.

Unlocking Large Scale Uncertainty Quantification with In Transit Iterative Statistics

“…Slightly different linear profiles can be proposed as γ(n) = 0.5[1+(n−1)/(N −1)]. Several tests on simple analytical functions (where the true quantile can be known) have been performed in order to calibrate and validate these γ-profiles [40,20]. Other algorithmic developments are currently under study to further improve the robustness of the Robbins-Monro estimate.…”

“…The goal is to reduce the data to store to disk and to avoid the time penalty to write and then read back the raw data set as required by the classical post hoc analysis approach. In recent works, we proposed the Melissa framework for the on-line data aggregation of high-resolution ensemble runs [46,40]. As soon as each available simulation provides the results produced to a set of staging nodes, these nodes process them to update the statistics on a first-come-first-served basis thanks to one-pass algorithms.…”