MPIWiz

Abstract

Message Passing Interface (MPI) is a widely used standard for managing coarse-grained concurrency on distributed computers. Debugging parallel MPI applications, however, has always been a particularly challenging task due to their high degree of concurrent execution and non-deterministic behavior. Deterministic replay is a potentially powerful technique for addressing these challenges, with existing MPI replay tools adopting either data-replay or order-replay approaches. Unfortunately, each approach has its tradeoffs. Data-replay generates substantial log sizes by recording every communication message. Order-replay generates small logs, but requires all processes to be replayed together. We believe that these drawbacks are the primary reasons that inhibit the wide adoption of deterministic replay as the critical enabler of cyclic debugging of MPI applications.This paper describes subgroup reproducible replay (SRR), a hybrid deterministic replay method that provides the benefits of both data-replay and order-replay while balancing their trade-offs. SRR divides all processes into disjoint groups. It records the contents of messages crossing group boundaries as in data-replay, but records just message orderings for communication within a group as in order-replay. In this way, SRR can exploit the communication locality of traffic patterns in MPI applications. During replay, developers can then replay each group individually. SRR reduces recording overhead by not recording intra-group communication, and reduces replay overhead by limiting the size of each replay group. Exposing these tradeoffs gives the user the necessary control for making deterministic replay practical for MPI applications.We have implemented a prototype, MPIWIZ, to demonstrate and evaluate SRR. MPIWIZ employs a replay framework that allows transparent binary instrumentation of both library and system calls. As a result, MPIWIZ replays MPI applications with no source code modification and relinking, and handles nondeterminism in both MPI and OS system calls. Our preliminary results show that MPIWIZ can reduce recording overhead by over a factor of four relative to data-replay, yet without requiring the entire application to be replayed as in order-replay. Recording increases execution time by 27% while the application can be replayed in just 53% of its base execution time.

A Survey of Graph Comparison Methods with Applications to Nondeterminism in High-Performance Computing

“…State-of-the-art record-and-replay tools such as ReMPI (Sato et al, 2015) target production-scale runs and prioritize scalability in terms of runtime and record size. Other record-and-replay tools target hybrid MPI + OpenMP executions (Budanur et al, 2012), MPI applications using one-sided communication (Qian et al, 2016b,a), replay of isolated subgroups of processes (Xue et al, 2009), and probabilistic replay (Park et al, 2009). In addition, tools such as NINJA (Sato et al, 2017) are used in conjunction with record-and-replay tools to improve the chances of capturing nondeterministic bugs.…”

Record-and-Replay Techniques for HPC Systems: A Survey

“…Xue et al proposed Subgroup Reproducible Replay (SRR) [54] to address not only the problem of trace size, but also provide the user with flexible replay options (i.e. replaying only those processes deemed relevant to the debugging task).…”