Transitioning from File-Based HPC Workflows to Streaming Data Pipelines with openPMD and ADIOS2

Abstract

This paper aims to create a transition path from file-based IO to streaming-based workflows for scientific applications in an HPC environment. By using the openPMP-api, traditional workflows limited by filesystem bottlenecks can be overcome and flexibly extended for in situ analysis. The openPMD-api is a library for the description of scientific data according to the Open Standard for Particle-Mesh Data (openPMD). Its approach towards recent challenges posed by hardware heterogeneity lies in the decoupling of data description in domain sciences, such as plasma physics simulations, from concrete implementations in hardware and IO. The streaming backend is provided by the ADIOS2 framework, developed at Oak Ridge National Laboratory. This paper surveys two openPMD-based loosely coupled setups to demonstrate flexible applicability and to evaluate performance. In loose coupling, as opposed to tight coupling, two (or more) applications are executed separately, e.g. in individual MPI contexts, yet cooperate by exchanging data. This way, a streaming-based workflow allows for standalone codes instead of tightlycoupled plugins, using a unified streaming-aware API and leveraging high-speed communication infrastructure available in modern compute clusters for massive data exchange. We determine new challenges in resource allocation and in the need of strategies for a flexible data distribution, demonstrating their influence on efficiency and scaling on the Summit compute system. The presented setups show the potential for a more flexible use of compute resources brought by streaming IO as well as the ability to increase throughput by avoiding filesystem bottlenecks.

From Compact Plasma Particle Sources to Advanced Accelerators with Modeling at Exascale

“…A highperformance, openPMD C++ and Python reference implementation, co-developed by LBNL and HZDR/CASUS, is used in massively parallel codes and in data analysis [27]. Recently, data streaming techniques were developed to transition traditional post-processing scripts to online, massively parallel data analysis workflows, which can be co-located with simulations and enable rapidly prototyping for scientific intransit analysis [28].…”

Julia as a unifying end-to-end workflow language on the Frontier exascale system

“…This instance demonstrates Julia's potential as a unified language that can seamlessly connect different stages of an HPC workflow, from computational simulation to data visualization. Future work includes further performance tuning and evaluation of the trade-offs for in-memory streaming data pipelines [34]. By bridging various components in a typical HPC workflow, Julia could emerge as a compelling option for scientists and engineers working on complex, large-scale computational problems.…”

Modeling of advanced accelerator concepts

“…A challenge arises in the rapid setup of in situ processing pipelines, since typical approaches of implementing a virtual detector in the simulation code itself can be more complicated compared to scripted (serial) analysis workflows on files. Innovative software design can potentially alleviate this challenge, with approaches such as streaming data with close-to-file-like analysis syntax [135] or embedding of interpreters into running simulations to execute scripts on in-memory data. Continued research in this area and close collaboration with computer science and engineering teams will be essential to ensure that flexible, science-case specific analysis can be designed by accelerator and beam physicists.…”