Complex scientific applications made fault-tolerant with the sparse grid combination technique

Abstract

Ultra-large-scale simulations via solving partial differential equations (PDEs) require very large computational systems for their timely solution. Studies shown the rate of failure grows with the system size, and these trends are likely to worsen in future machines. Thus, as systems, and the problems solved on them, continue to grow, the ability to survive failures is becoming a critical aspect of algorithm development. The sparse grid combination technique (SGCT) which is a cost-effective method for solving higher dimensional PDEs can be easily modified to provide algorithm-based fault tolerance. In this article, we describe how the SGCT can produce fault-tolerant versions of the Gyrokinetic Electromagnetic Numerical Experiment plasma application, Taxila Lattice Boltzmann Method application, and Solid Fuel Ignition application. We use an alternate component grid combination formula by adding some redundancy on the SGCT to recover data from lost processes. User-level failure mitigation (ULFM) message passing interface (MPI) is used to recover the processes, and our implementation is robust over multiple failures and recovery (processes and nodes). An acceptable degree of modification of the applications is required. Results using the 2-D SGCT show competitive execution times with acceptable error (within 0.1% to 1.0%), compared to the same simulation with a single full resolution grid. The benefits improve when the 3-D SGCT is used. Experiments show the applications ability to successfully recover from multiple failures, and applying multiple SGCT reduces the computed solution error. Process recovery via ULFM MPI increases from approximately 1.5 sec at 64 cores to approximately 5 sec at 2048 cores for a one-off failure. This compares applications' built-in checkpointing with job restart in conjunction with the classical SGCT on failure, which have overheads four times as large for a single failure, excluding the recomputation overhead. An analysis for a long-running application considering recomputation times indicates a reduction in overhead of over an order of magnitude.

Exploring Parallel MPI Fault Tolerance Mechanisms for Phylogenetic Inference with RAxML-NG

“…There already exist some fault-tolerant scientific applications. For example Ali et al (2016) implemented a faulttolerant numeric linear equation and partial equation solver. Obersteiner et al (2017) extended a plasma simulation, Laguna et al (2016) a molecular dynamics simulation, and Engelmann and Geist (2003) a Fast Fourier Transformation that gracefully handle hardware faults.…”

“…Researchers have already used ULFM for other scientific software (Ali et al, 2016;Engelmann and Geist, 2003;Kohl et al, 2017;Laguna et al, 2016;Obersteiner et al, 2017). ULFM reports failures by returning an error on at least one rank which participated in the failed communication.…”

Fault tolerance of MPI applications in exascale systems: The ULFM solution

“…The operation MPIX Comm failure ack enables users to acknowledge all locally notified failures in the communication context. 1 When using unnamed communications, this routine pro-vides the application a way to resume any-source operations, as long as the list of failed processes does not change.…”

“…Ali et al [1] focus on non-shrinking recovery of PDE-based applications, re-spawning replacement processes on the same node, when they are still available, or otherwise in pre-allocated spare-nodes. As in their shrinking proposal [47], they use the SGCT Algorithm-Based Fault Tolerance (ABFT) strategy to approximate recovery of multiple failures, rather than the exact recovery through checkpointing.…”

“…Besides, the spatially coordinated checkpointing protocol applied by CPPC reduces the log size, the log memory requirements, and overall the resilience impact on the applications. Local forward Recomputation is not needed 1 In some cases, it is enough to notify a subset of live processes 2 Or activation of warm spare processes…”