Local rollback for resilient MPI applications with application-level checkpointing and message logging

Losada, Nuria Riopérez; Bosilca, George; Bouteiller, Aurélien; González, Patricia; Martín, María J.

doi:10.1016/j.future.2018.09.041

Cited by 23 publications

(23 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A local rollback protocol that can be generally applied to single program, multiple data (SPMD) applications is proposed in [35]. It combines the ComPiler for Portable Checkpointing (CPPC) tool, message logging, and ULFM.…”

Section: Non-shrinking Solutionsmentioning

confidence: 99%

Fault tolerance of MPI applications in exascale systems: The ULFM solution

Losada

González

Martín

et al. 2020

Future Generation Computer Systems

Self Cite

View full text Add to dashboard Cite

The growth in the number of computational resources used by high-performance computing (HPC) systems leads to an increase in failure rates. Fault-tolerant techniques will become essential for long-running applications executing in future exascale systems, not only to ensure the completion of their execution in these systems but also to improve their energy consumption. Although the Message Passing Interface (MPI) is the most popular programming model for distributed-memory HPC systems, as of now, it does not provide any fault-tolerant construct for users to handle failures. Thus, the recovery procedure is postponed until the application is aborted and re-spawned. The proposal of the User Level Failure Mitigation (ULFM) interface in the MPI forum provides new opportunities in this field, enabling the implementation of resilient MPI applications, system runtimes, and programming language constructs able to detect and react to failures without aborting their execution. This paper presents a global overview of the resilience interfaces provided by the ULFM specification, covers archetypal usage patterns and building blocks, and surveys the wide variety of application-driven solutions that have exploited them in recent years. The large and varied number of approaches in the literature proves that ULFM provides the necessary flexibility to implement efficient faulttolerant MPI applications. All the proposed solutions are based on application-driven recovery mechanisms, which allows reducing the overhead and obtaining the required level of efficiency needed in the future exascale platforms.

show abstract

Section: Non-shrinking Solutionsmentioning

confidence: 99%

Fault tolerance of MPI applications in exascale systems: The ULFM solution

Losada

González

Martín

et al. 2020

Future Generation Computer Systems

Self Cite

View full text Add to dashboard Cite

show abstract

“…Buddy in-memory checkpointing can be a limiting aspect as it does not cover multiple-node failure scenarios, but it still provides efficient resilience as these scenarios are unlikely. This approach also makes it possible to adopt a local restart strategy such as [13], with computation kept running on the surviving nodes. This is made possible thanks to the ULFM MPI proposal implementation [5] and by using message logging [3], [7].…”

Section: Context and Related Workmentioning

confidence: 99%

From tasks graphs to asynchronous distributed checkpointing with local restart

Lion

Thibault

2020

2020 IEEE/ACM 10th Workshop on Fault Tolerance for HPC at eXtreme Scale (FTXS)

View full text Add to dashboard Cite

The ever-increasing number of computation units assembled in current HPC platforms leads to a concerning increase in fault probability. Traditional checkpoint/restart strategies avoid wasting large amounts of computation time when such fault occurs. With the increasing amount of data dealt with by current applications, these strategies however suffer from their data transfer demand becoming unreasonable, or the entailed global synchronizations. Meanwhile, the current trend towards task-based programming is an opportunity to revisit the principles of the checkpoint/restart strategies. We here propose a checkpointing scheme which is closely tied to the execution of task graphs. We describe how it allows for completely asynchronous and distributed checkpointing, as well as localized node restart, thus opening up for very large scalability. We also show how a synergy between the application data transfers and the checkpointing transfers can lead to a reasonable additional network load, measured to be lower than +10% on a dense linear algebra example.

show abstract

“…Implementations of causal message logging protocols in MPI libraries include MPICH-V [7], MPICH-V2 [10], and Charm++ [19]. The work of Losada et al [18] implements the most recent variation on pessimistic message logging protocols; it employs the VProtocol in Open MPI 1 , and a source-to-source compiler called CPPC. Our work, in contrast to pessimistic, optimistic, and causal message logging protocols, entirely disables event logging; we statically analyse the underlying applications, and guarantee the absence of non-deterministic events that need to be replayed.…”

Section: Related Workmentioning

confidence: 99%

Implementing efficient message logging protocols as MPI application extensions

Dichev

Nikolopoulos

2019

Proceedings of the 26th European MPI Users' Group Meeting

View full text Add to dashboard Cite

Message logging protocols are enablers of local rollback, a more efficient alternative to global rollback, for fault tolerant MPI applications. Until now, message logging MPI implementations have incurred the overheads of a redesign and redeployment of an MPI library, as well as continued performance penalties across various kernels. Successful research efforts for message logging implementations do exist, but not a single one of them can be easily deployed today by more than a few experts. In contrast, in this work we build efficient message logging capabilities on top of an MPI library with no message logging capabilities; we do so for two different HPC kernels, one with a global exchange pattern (CG), and one with a neighbourhood exchange pattern (LULESH). While our library of choice ULFM detects failure and recovers MPI communicators, we build on that to then restore the intra-and inter-process data consistency of both applications. This task turns out to be challenging, and we present the methodology for doing so in this work. In the end, we achieve message logging capabilities for each kernel, without the need for an actual message logging runtime underneath. On the performance side, we match state-of-the-art solutions and (a) eliminate event logging and the event logger component altogether, and (b) design a hybrid protocol, which gracefully shifts between global and local rollback, depending on the available payload logging memory. Such a hybrid protocol between local and global rollback has not been previously proposed to our knowledge. Our extensions span a few hundred lines of code for each kernel, are open-sourced, and enable local and global rollback after process failure.

show abstract

Local rollback for resilient MPI applications with application-level checkpointing and message logging

Cited by 23 publications

References 14 publications

Fault tolerance of MPI applications in exascale systems: The ULFM solution

Fault tolerance of MPI applications in exascale systems: The ULFM solution

From tasks graphs to asynchronous distributed checkpointing with local restart

Implementing efficient message logging protocols as MPI application extensions

Contact Info

Product

Resources

About