In Situ Statistical Analysis for Parametric Studies

Abstract

In situ processing proposes to reduce storage needs and I/O traffic by processing results of parallel simulations as soon as they are available in the memory of the compute processes. We focus here on computing in situ statistics on the results of N simulations from a parametric study. The classical approach consists in running various instances of the same simulation with different values of input parameters. Results are then saved to disks and statistics are computed post mortem, leading to very I/O intensive applications. Our solution is to develop Melissa, an in situ library running on staging nodes as a parallel server. When starting, simulations connect to Melissa and send the results of each time step to Melissa as soon as they are available. Melissa implements iterative versions of classical statistical operations, enabling to update results as soon as a new time step from a simulation is available. Once all statistics ar updated, the time step can be discarded. We also discuss two different approaches for scheduling simulation runs: the jobs-in-job and the multi-jobs approaches. Experiments run instances of the Computational Fluid Dynamics Open Source solver Code_Saturne. They confirm that our approach enables one to avoid storing simulation results to disk or in memory.

Large scale in transit computation of quantiles for ensemble runs

“…However, they all rely on classical non-iterative algorithms, requiring to accumulate first all simulation results in file or memory if doable. Based on a different architecture, the Melissa framework [17,16] has been recently proposed for the on-line data aggregation of high resolution ensemble runs. Other in situ processing frameworks (see [18,19,20,21,22]) enable in situ and in transit processing but for a single simulation.…”