Post-failure recovery of MPI communication capability

Bland, Wesley; Bouteiller, Aurélien; Hérault, Thomas; Bosilca, George; Dongarra, Jack

doi:10.1177/1094342013488238

Cited by 164 publications

(121 citation statements)

References 29 publications

Supporting

Mentioning

110

Contrasting

Unclassified

Order By: Relevance

“…In this paper, we tackle this issue from a different perspective, with the goal of improving the efficiency of the existing implementation of the fault tolerant constructs added to the Message Passing Interface (MPI) by the User-Level Failure Mitigation (ULFM) [3] proposal. Moreover, MPI being a de-facto parallel programming paradigm, one of our main concerns will be the time-to-solution, more specifically the scalability, of the proposed agreement.…”

Section: Use Case: the Ulfm Agreementmentioning

confidence: 99%

Practical scalable consensus for pseudo-synchronous distributed systems

Hérault

Bouteiller

Bosilca

et al. 2015

Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

Self Cite

View full text Add to dashboard Cite

The ability to consistently handle faults in a distributed environment requires, among a small set of basic routines, an agreement algorithm allowing surviving entities to reach a consensual decision between a bounded set of volatile resources. This paper presents an algorithm that implements an Early Returning Agreement (ERA) in pseudo-synchronous systems, which optimistically allows a process to resume its activity while guaranteeing strong progress. We prove the correctness of our ERA algorithm, and expose its logarithmic behavior, which is an extremely desirable property for any algorithm which targets future exascale platforms. We detail a practical implementation of this consensus algorithm in the context of an MPI library, and evaluate both its efficiency and scalability through a set of benchmarks and two fault tolerant scientific applications.

show abstract

Section: Use Case: the Ulfm Agreementmentioning

confidence: 99%

Practical scalable consensus for pseudo-synchronous distributed systems

Hérault

Bouteiller

Bosilca

et al. 2015

Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

Self Cite

View full text Add to dashboard Cite

show abstract

“…Mini-Ckpts is a known example of a framework that emphasizes the recovery of the OS environment by preserving kernel structures in persistent memory [31]. Similarly, the ULFM MPI provides recovery of the communication environment from the failure of processes by reconstructing the MPI communicator by creating consensus among the remaining set of processes [7].…”

Section: Environment State Patternmentioning

confidence: 99%

“…While the pattern seeks to restructure the sub-systems in an operating state that is functionally equivalent to the fault-free state, the pattern may result in the operation of the system in degraded condition, which incurs additional time overhead to the system. Existing solutions that restructure the system in response to an event include the ULFM extension to the MPI standard [7], which allows parallel applications to get notifications of process failures. ULFM provides a set of routines to revoke and restructure a MPI communicator that consists of the remaining active processes.…”

Section: Restructure Patternmentioning

confidence: 99%

Resilience Design Patterns: A Structured Approach to Resilience at Extreme Scale

Hukerikar

Engelmann

2017

JSFI

View full text Add to dashboard Cite

Reliability is a serious concern for future extreme-scale high-performance computing (HPC) systems. Projections based on the current generation of HPC systems and technology roadmaps suggest the prevalence of very high fault rates in future systems. While the HPC community has developed various resilience solutions, application-level techniques as well as system-based solutions, the solution space remains fragmented. There are no formal methods and metrics to integrate the various HPC resilience techniques into composite solutions, nor are there methods to holistically evaluate the adequacy and efficacy of such solutions in terms of their protection coverage, and their performance & power efficiency characteristics. Additionally, few of the current approaches are portable to newer architectures and software environments that will be deployed on future systems. In this paper, we develop a structured approach to the design, evaluation and optimization of HPC resilience using the concept of design patterns. A design pattern is a general repeatable solution to a commonly occurring problem. We identify the problems caused by various types of faults, errors and failures in HPC systems and the techniques used to deal with these events. Each well-known solution that addresses a specific HPC resilience challenge is described in the form of a pattern. We develop a complete catalog of such resilience design patterns, which may be used by system architects, system software and tools developers, application programmers, as well as users and operators as essential building blocks when designing and deploying resilience solutions. We also develop a design framework that enhances a designer's understanding the opportunities for integrating multiple patterns across layers of the system stack and the important constraints during implementation of the individual patterns. It is also useful for defining mechanisms and interfaces to coordinate flexible fault management across hardware and software components. The resilience patterns and the design framework also enable exploration and evaluation of design alternatives and support optimization of the cost-benefit trade-offs among performance, protection coverage, and power consumption of resilience solutions. The overall goal of this work is to establish a systematic methodology for the design and evaluation of resilience technologies in extreme-scale HPC systems that keep scientific applications running to a correct solution in a timely and cost-efficient manner despite frequent faults, errors, and failures of various types.

show abstract

“…User-Level Failure Mitigation (ULFM) [22] is the proposal currently being discussed and iteratively refined at the MPI Forum. Although we can find implementations supporting it experimentally, it cannot be considered MPI compliant, and its contents are likely to change before its possible final incorporation into a future version of the Standard.…”

Section: Mpi Resiliencementioning

confidence: 99%

Supporting automatic recovery in offloaded distributed programming models through MPI-3 techniques

Peña

Beltrán

Clauss

et al. 2017

Proceedings of the International Conference on Supercomputing

View full text Add to dashboard Cite

In this paper we describe the design of fault tolerance capabilities for general-purpose offload semantics, based on the OmpSs programming model. Using ParaStation MPI, a production MPI-3.1 implementation, we explore the features that, being standard compliant, an MPI stack must support to provide the necessary fault tolerance guarantees, based on MPI's dynamic process management. Our results, including synthetic benchmarks and applications, reveal low runtime overhead and efficient recovery, demonstrating that the existing MPI standard provided us with sufficient mechanisms to implement an effective and efficient fault-tolerant solution.

show abstract

Post-failure recovery of MPI communication capability

Cited by 164 publications

References 29 publications

Practical scalable consensus for pseudo-synchronous distributed systems

Practical scalable consensus for pseudo-synchronous distributed systems

Resilience Design Patterns: A Structured Approach to Resilience at Extreme Scale

Supporting automatic recovery in offloaded distributed programming models through MPI-3 techniques

Contact Info

Product

Resources

About