In situ visualization of large-scale turbulence simulations in Nek5000 with ParaView Catalyst

Abstract

In situ visualization on high-performance computing systems allows us to analyze simulation results that would otherwise be impossible, given the size of the simulation data sets and offline post-processing execution time. We develop an in situ adaptor for Paraview Catalyst and Nek5000, a massively parallel Fortran and C code for computational fluid dynamics. We perform a strong scalability test up to 2048 cores on KTH’s Beskow Cray XC40 supercomputer and assess in situ visualization’s impact on the Nek5000 performance. In our study case, a high-fidelity simulation of turbulent flow, we observe that in situ operations significantly limit the strong scalability of the code, reducing the relative parallel efficiency to only $$\approx 21\%$$ ≈ 21 % on 2048 cores (the relative efficiency of Nek5000 without in situ operations is $$\approx 99\%$$ ≈ 99 % ). Through profiling with Arm MAP, we identified a bottleneck in the image composition step (that uses the Radix-kr algorithm) where a majority of the time is spent on MPI communication. We also identified an imbalance of in situ processing time between rank 0 and all other ranks. In our case, better scaling and load-balancing in the parallel image composition would considerably improve the performance of Nek5000 with in situ capabilities. In general, the result of this study highlights the technical challenges posed by the integration of high-performance simulation codes and data-analysis libraries and their practical use in complex cases, even when efficient algorithms already exist for a certain application scenario.

Understanding the Impact of Synchronous, Asynchronous, and Hybrid In-Situ Techniques in Computational Fluid Dynamics Applications

“…Although the Nek5000 scales well, the execution time to generate images with ParaView/Catalyst does not scale and remains nearly constant. The MPI collective communication was identified as the bottleneck in the previous study [4]. This also corresponds to our poorly scaling overall execution time for image generation as the number of cores increased.…”

“…However, because of the required collective communications, the visualization task frequently scales much less than the simulation. As a result, synchronous in-situ approaches can be problematic with the MPI collective communication, as shown by Atzori et al [4]. Using the asynchronous in-situ approach, it is thus advantageous to assign a smaller set of resources to the in-situ task than to the simulation.…”

“…a) Implementation: In the synchronous image generation case, we used the in-situ adaptor functions from the in-situ package repository developed by Atzori et al [4] as the Nekproc adaptor functions in Fig. 2 to connect the Fortranbased Nek5000 and the C/C++-based in-situ task.…”

Higgs Boson Classification: Brain-inspired BCPNN Learning with StreamBrain

“…ParaView is an open-source, multi-platform data analysis and visualization application, which has been used to provide in-situ visualization of many different HPC scientific simulations [20]. In our case, that HPC simulation is the receptive field of the HCUs.…”