Reinit$$^{++}$$: Evaluating the Performance of Global-Restart Recovery Methods for MPI Fault Tolerance

Abstract

Scaling supercomputers comes with an increase in failure rates due to the increasing number of hardware components. In standard practice, applications are made resilient through checkpointing data and restarting execution after a failure occurs to resume from the latest checkpoint. However, redeploying an application incurs overhead by tearing down and reinstating execution, and possibly limiting checkpointing retrieval from slow permanent storage. In this paper we present Reinit ++ , a new design and implementation of the Reinit approach for global-restart recovery, which avoids application re-deployment. We extensively evaluate Reinit ++ contrasted with the leading MPI fault-tolerance approach of ULFM, implementing globalrestart recovery, and the typical practice of restarting an application to derive new insight on performance. Experimentation with three different HPC proxy applications made resilient to withstand process and node failures shows that Reinit ++ recovers much faster than restarting, up to 6×, or ULFM, up to 3×, and that it scales excellently as the number of MPI processes grows.

Resilience and fault tolerance in high-performance computing for numerical weather and climate prediction

“…MPI_ReInit (Georgakoudis et al, 2020; Laguna et al, 2016) is a similar approach, but it does not expose any low-level APIs like MPI-ULFM. Instead, it focuses on efficient online rollback recovery, simplifying the low-level fault detection and notification mechanism accommodated by MPI-ULFM.…”

“…MPI_ReInit does not specify any data recovery schemes, allowing the use of external software. The latest implementation (Georgakoudis et al, 2020) demonstrates better scalability than MPI-ULFM in the absence of failures. On data recovery for MPI programs, Global View Resilience (GVR, Chien et al, 2015) and VeloC (Nicolae et al, 2019) accommodate generic APIs for data persistency.…”

MATCH: An MPI Fault Tolerance Benchmark Suite

“…Note that we choose to evaluate different fault tolerance techniques by triggering a process failure, which does not mean that the MPI recovery frameworks do not support recovery from a node failure. On the one hande Reinit can recover from a node failure [14], on the other hand the current ULFM implementation cannot. In our case, it is sufficient to evaluate on MPI process failures to compare the performance difference when using FTI checkpointing in ULFM and Reinit.…”

“…To implement FTI with Reinit, the only thing to notice is to move the FTI Init() and FTI Finalize() functions into the resilient main() function as well. Please read work on Reinit [13], [14], [19] for the design and implementation details of Reinit.…”

“…Later, because of the inefficiency of restarting an application, HPC practitioners propose MPI recovery mechanisms to restore the MPI state online. User-level Fault Mitigation (ULFM) [11] and Reinit [12], [13], [14] are the two pioneering MPI recovery frameworks in this effort. ULFM supports a wide range of recovery strategies, including local forward recovery and global-restart recovery, whereas Reinit only supports global-restart recovery.…”