In-situ visualization and computational steering for large-scale simulation of turbulent flows in complex geometries

Abstract

Large-scale simulations conducted on supercomputers such as leadership-class computing facilities allow researchers to simulate and study complex problems with high fidelity, and thus have become indispensable in diverse areas of science and engineering. These high-fidelity simulations generate vast amount of data which is becoming more and more difficult to transform into knowledge using traditional visual analysis approaches. For instance, there are tremendous challenges in analyzing big data produced by high-fidelity simulations in order to gain meaningful insight into complex phenomena such as turbulent two-phase flows. The traditional workflow, which consists in conducting simulations on supercomputers and recording enormous raw simulation data to disk for further post-processing and visualization, is no longer a viable approach due to prohibitive cost of disk access and considerable amount of time spent on data transfer. Visual Analytics approaches for big data have to be researched and employed to address the problem of knowledge discovery from such large-scale simulations. One approach to tackle this issue is to couple a numerical simulation with in-situ visualization so that the post-processing and visualization occurs while the simulation is running. This in-situ approach minimizes data storage by extracting and visualizing important features of the data directly within the simulation without saving the raw data to disk. In addition, in-situ visualization allows users to steer the simulation by adjusting input parameters while the simulation is ongoing. In this paper, we present our approach for in-situ visualization of simulation data generated by massively parallel finite-element computational fluid dynamics solver (PHASTA) instrumented and linked with ParaView Catalyst. We demonstrate our in-situ visualization and simulation steering capability with a fully resolved turbulent flow through 2×2 reactor subchannel complex geometry. In addition, we present results from our insitu visualization for turbulent flow simulations conducted on the supercomputers Cray XK7 "Titan" at Oak Ridge National Laboratory and IBM BlueGene/Q "Mira" at Argonne National Laboratory up to 32,768 cores and examine the overhead of insitu visualization and its effect on code performance.

Software tools to enable immersive simulation

“…Our work builds directly upon previous implementations of Catalyst in the flow solver PHASTA [54,4]. Where earlier demonstrations of co-processing functioned with Catalyst hooked directly into the PHASTA solver, our work differs in that we implement SENSEI to interface between the solver and Catalyst.…”

“…A powerful extension of co-processing is computational steering, where users can take advantage of real time simulation feedback to rapidly explore the design space and extract insight. Computational steering has been previously demonstrated in the flow solver PHASTA with Catalyst [50,54], though steering was accomplished via live edits of a solver parameter file, and geometric deformations were not implemented. More recently, Catalyst has been used to model turbidity currents [9], where steering is introduced through application specific LibMesh-sed-imentation code on solver parameters such as time step and tolerances.…”

SLoG: Large-Scale Logging Middleware for HPC and Big Data Convergence

“…Analyzing these events while the application is running can be very useful, for instance to verify that the application behaves as expected, or to run in-situ visualization [21]. This information can also be used for in-situ analytics [22] or computational steering [23]- [25]. The latter can in turn influence the simulation in real-time, e.g.…”

In situ visualization of large-scale turbulence simulations in Nek5000 with ParaView Catalyst

“…With extreme-scale on the horizon, Ma [15] presented the challenges and opportunities of in situ visualization, later realized by Rasquin et al [23], combining both in situ visualization with computational steering, running the flow solver PHASTA on 160k cores, connected to ParaView running on a separate visualization cluster. More recently, using Catalyst, Yi et al [29] demonstrated that both simulations, visualization, and steering could be executed on the same computational resources. The feasibility of extreme-scale in situ processing was later demonstrated by Ayachit et al [2], running PHASTA using SENSEI and Catalyst for in situ visualization on more than 1 million MPI ranks, achieving a low 13% in situ overhead.…”