Record-and-Replay Techniques for HPC Systems: A Survey

Abstract

Record-and-replay techniques provide the ability to record executions of nondeterministic applications and re-execute them identically. These techniques find use in the contexts of debugging, reproducibility, and fault-tolerance, especially in the presence of nondeterministic factors such as message races. Record-and-replay techniques are highly diverse in terms of the fidelity of replay they provide, the assumptions they make about the recorded application, the programming models they target, and the runtime overheads they impose. In the high performance computing (HPC) environment, all the above factors must be considered in concert, thus presenting additional implementation challenges. In this manuscript, we survey record-and-replay techniques in terms of the programming models they target and the workloads on which they were evaluated, providing a categorization of these techniques benefiting application developers and researchers targeting exascale challenges. This manuscript answers three questions through this survey: What are the gaps in the existing space of record-and-replay techniques? What is the roadmap to widespread use of record-and-replay on production-scale HPC workloads? And, what are the critical open problems that must be addressed to make record-and-replay viable at exascale?

A Survey of Graph Comparison Methods with Applications to Nondeterminism in High-Performance Computing

“…In production HPC environments, record-and-replay tools allow users to record a nondeterministic application's execution and then replay it exactly, thus enabling the reproducibility of nondeterministic bugs (Chapp et al, 2018). State-of-the-art record-and-replay tools such as ReMPI (Sato et al, 2015) target production-scale runs and prioritize scalability in terms of runtime and record size.…”

“…The convergence of extreme hardware concurrency and the effective overlap of computation and communication in asynchronous executions are resulting in growing nondeterminism in High-Performance Computing (HPC) applications, as illustrated in Figure 1 and presented in Ahn et al (2013); Gopalakrishnan et al (2017); Sato et al (2017); Chapp et al (2015Chapp et al ( , 2018Chapp et al ( , 2021. Nondeterminism can manifest at multiple levels in the software stack: it can manifest in lowlevel communication primitives (e.g., the inherent nondeterminism of nonblocking matching functions in MPI); it can manifest in libraries (e.g., dynamic load-balancing libraries), as presented in Lusk et al (2015); or it can display at the application level (e.g., Monte-Carlo simulations).…”

“…Throughout this article, we discuss the literature on these categories of graph comparison using the Chapp-Taufer format (Chapp et al, 2018) that maps publications based on publication time and categories. A box represents each article; color represents different solutions (i.e., general or specific); and symbols represent algorithms features.…”

Building a Vision for Reproducibility in the Cyberinfrastructure Ecosystem: Leveraging Community Efforts

“…While capture of intermediate data products can potentially bolster efforts to achieve reproducibility, doing so necessarily comes at the cost of scalability, especially in the HPC setting. Efforts to achieve scalable record and replay of HPC applications indicate that capturing fine-grained data about the intermediate state of parallel executions remains an active and challenging area of research [13]. Hence, in our view the feasibility of Recommendation 4-1's guidelines regarding capture of intermediate data must be managed on a case-by-case basis.…”

“…For multiple parallel runtimes, to cope with evolving HPC system architectures, the use of multiple parallel runtimes (e.g., MPI + OpenMP) in a single codebase has become increasingly common in scientific computing. The effect of mixing these runtimes on application-level nondeterminism has been identified as a major challenge in the push to exascale [26], and the scarcity of tools for mitigating non-determinism in these types of codebases has been documented [13].…”