In situ and in-transit analysis of cosmological simulations

Abstract

Modern cosmological simulations have reached the trillion-element scale, rendering data storage and subsequent analysis formidable tasks. To address this circumstance, we present a new MPI-parallel approach for analysis of simulation data while the simulation runs, as an alternative to the traditional workflow consisting of periodically saving large data sets to disk for subsequent 'offline' analysis. We demonstrate this approach in the compressible gasdynamics/N-body code Nyx, a hybrid MPI + OpenMP code based on the BoxLib framework, used for large-scale cosmological simulations. We have enabled on-the-fly workflows in two different ways: one is a straightforward approach consisting of all MPI processes periodically halting the main simulation and analyzing each component of data that they own ('in situ'). The other consists of partitioning processes into disjoint MPI groups, with one performing the simulation and periodically sending data to the other 'sidecar' group, which post-processes it while the simulation continues ('in-transit'). The two groups execute their tasks asynchronously, stopping only to synchronize when a new set of simulation data needs to be analyzed. For both the in situ and in-transit approaches, we experiment with two different analysis suites with distinct performance behavior: one which finds dark matter halos in the simulation using merge trees to calculate the mass contained within iso-density contours, and another which calculates probability distribution functions and power spectra of various fields in the simulation. Both are common analysis tasks for cosmology, and both result in summary statistics significantly smaller than the original data set. We study the behavior of each type of analysis in each workflow in order to determine the optimal configuration for the different data analysis algorithms.

Adaptive Configuration of In Situ Lossy Compression for Cosmology Simulations via Fine-Grained Rate-Quality Modeling

“…Note that halo is further categorized into many small halos and few large halos, only the former can be false-detected under high error bound. In general, we intend to concentrate and preserve information for large halos based on cosmological analysis [12]. In fact, we also observe almost no halo position change under even extreme compression during our experiment.…”

“…Then, there is the Most Bound Particle, which is defined as the particle within a halo with the lowest potential. For the Nyx simulation, which is an Eulerian simulation instead of Lagrangian simulation, the Halo Finding algorithm uses density data to identify halos [12]. For decompressed data, some of the information can be distorted from the original.…”

“…Large-scale scientific simulations running with leadership supercomputers are essential in many science and engineering domains such as cosmology studies. Modern cosmological simulations are used by researchers and scientists to investigate new fundamental astrophysics ideas, develop and evaluate new cosmological probes, assist large-scale cosmological surveys, and investigate systematic uncertainties [12,19]. Historically such studies have required large simulations that are highly computation and storage intensive, which are run on leadership supercomputers.…”

Opportunities for Cost Savings with In-Transit Visualization

“…They find there were times when in-transit analysis was faster due to how the analysis code scaled. Friesen et al [5] describes a setup where in-line and in-transit visualization are used in conjunction with a cosmological code to run two different analysis routines. They analyze the time to solution using both in situ techniques, finding that there are configurations where in-transit is faster to use, due to the inter-node communication overhead of the analysis routines.…”

Accelerating In-Transit Co-Processing for Scientific Simulations Using Region-Based Data-Driven Analysis

“…This is one of the most pressing matters for large-scale scientific simulations. Simulations can result in multiple tera-or petabytes of generated data, making it challenging to save all necessary data to permanent storage due to limited storage capacity or time-consuming I/O operations [3][4][5][6]. In-situ or in-transit visualization and analysis are often used for each time step while the data is generated.…”