Application Level Fault Recovery: Using Fault-Tolerant Open MPI in a PDE Solver

Ali, Md. Mohsin; Southern, James; Strazdins, Peter; Harding, Brendan

doi:10.1109/ipdpsw.2014.132

Cited by 24 publications

(22 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is tolerated as follows. Before the SGCT algorithm is applied, the loss of some processes in P i is detected using ULFM MPI (see [16] for details). Replacement processes are then created (with the same process grid size as P i ) on the same node when node failure is not happened.…”

Section: A Sgct Algorithm Overview and Process Organizationmentioning

confidence: 99%

“…In order to tolerate these failures, the faulty communicator is reconstructed (line 15, details are in [16]). Then the fault-tolerant SGCT is applied using the communicator W , with the combined solution u c I being used to re-initialize the sub-grid solutions {u i } (lines [16][17].…”

Section: B Integration Of the Sgct Algorithm Into Higherdimensional mentioning

confidence: 99%

“…An SGCT-based, fault-tolerant 2D advection equation solver capable of surviving single or multiple real process failures was proposed in [16]. This paper described in detail how ULFM MPI could be used for process recovery, and our work uses the same method.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

A fault-tolerant gyrokinetic plasma application using the sparse grid combination technique

Ali

Strazdins

Harding

et al. 2015

2015 International Conference on High Performance Computing &Amp; Simulation (HPCS)

Self Cite

View full text Add to dashboard Cite

Applications performing ultra-large scale simulations via solving PDEs require very large computational systems for their timely solution. Studies have shown the rate of failure grows with the system size and these trends are likely to worsen in future machines as less reliable components are used to reduce the energy cost. Thus, as systems, and the problems solved on them, continue to grow, the ability to survive failures is becoming a critical aspect of algorithm development. The sparse grid combination technique (SGCT) is a cost-effective method for solving time-evolving PDEs, especially for higherdimensional problems. It can also be easily modified to provide algorithm-based fault tolerance for these problems. In this paper, we show how the SGCT can produce a fault-tolerant version of the GENE gyrokinetic plasma application, which evolves a 5D complex density field over time. We use an alternate component grid combination formula to recover data from lost processes. User Level Failure Mitigation (ULFM) MPI is used to recover the processes, and our implementation is robust over multiple failures and recovery for both process and node failures. An acceptable degree of modification of the application is required. Results using the SGCT on two of the fields' dimensions show competitive execution times with acceptable error (within 0.1%), compared to the same simulation with a single full resolution grid. The benefits improve when the SGCT is used over three dimensions. Our experiments show that the GENE application can successfully recover from multiple process failures, and applying the SGCT the corresponding number of times minimizes the error for the lost sub-grids. Application recovery overhead via ULFM MPI increases from ∼1.5s at 64 cores to ∼5s at 2048 cores for a oneoff failure. This compares favourably to using GENE's in-built checkpointing with job restart in conjunction with the classical SGCT on failure, which have overheads four times as large for a single failure, excluding the backtrack overhead. An analysis for a long-running application taking into account checkpoint backtrack times indicates a reduction in overhead of over an order of magnitude.

show abstract

Section: A Sgct Algorithm Overview and Process Organizationmentioning

confidence: 99%

Section: B Integration Of the Sgct Algorithm Into Higherdimensional mentioning

confidence: 99%

See 1 more Smart Citation

A fault-tolerant gyrokinetic plasma application using the sparse grid combination technique

Ali

Strazdins

Harding

et al. 2015

2015 International Conference on High Performance Computing &Amp; Simulation (HPCS)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Although generally exhibiting excellent performance and resiliency, ABFT requires that the algorithm is innately able to incorporate fault tolerance and therefore might be a less generalist approach. In abft applications that require the restoration of a full set of processes [1], the recovery procedure for the MPI layer actually has strikingly similar requirement to the deployment of coordinated checkpoint with in-place restart. Unlike checkpoint/restart, this is not only an optimization, but a hard requirement.…”

Section: Forward Recovery With Complex Patternsmentioning

confidence: 99%

“…High Performance Computing, as observed by the Top 500 ranking, 1 has exhibited a constant progression of the computing power by a factor of two every 18 months for the last 15 years and the pace of progress has been only slightly disturbed by the financial turmoil in 2008. Following the long-term trend, the Exaflops milestone should be reached as soon as 2022.…”

Section: Introductionmentioning

confidence: 99%

Fault-Tolerant MPI

Bouteiller

2015

Computer Communications and Networks

View full text Add to dashboard Cite

As supercomputers are entering an era of massive parallelism where the frequency of faults is increasing, the MPI standard remains distressingly vague on the consequence of failures on MPI communications. In this chapter, we present the spectrum of techniques that can be applied to enable MPI application recovery, ranging from fully automatic to completely user driven. First, we present the effective deployment of most advanced checkpoint/restart techniques within the MPI implementation, so that failed processors are automatically restarted in a consistent state with surviving processes, at a performance cost. Then, we investigate how MPI can support application-driven recovery techniques, and introduce a set of extensions to MPI that allow restoring communication capabilities, while maintaining the extreme level of performance to which MPI users have become accustomed.

show abstract

Tree‐based fault‐tolerant collective operations for MPI

Margolin

Barak

2020

Concurrency and Computation

View full text Add to dashboard Cite

Summary With the increase in size and complexity of high‐performance computing systems, the probability of failures, and the cost of recovery grow. Parallel applications running on these systems should be able to continue running in spite of node failures at arbitrary times. Collective operations are essential for many parallel MPI applications, and are often the first to detect such failures. This work presents tree‐based fault‐tolerant collective operations, which combine fault detection and recovery as an integral part each operation. We do this by extending existing tree‐based algorithms, to allow for a collective operation to succeed despite failing nodes before or during its run. This differs from other approaches, where recovery takes place after a failure of such operations have failed. The article includes a comparison between the performance of the proposed algorithm and other approaches, as well as a simulator‐based analysis of performance at scale.

show abstract

Application Level Fault Recovery: Using Fault-Tolerant Open MPI in a PDE Solver

Cited by 24 publications

References 9 publications

A fault-tolerant gyrokinetic plasma application using the sparse grid combination technique

A fault-tolerant gyrokinetic plasma application using the sparse grid combination technique

Fault-Tolerant MPI

Tree‐based fault‐tolerant collective operations for MPI

Contact Info

Product

Resources

About