Fault tolerance of MPI applications in exascale systems: The ULFM solution

Abstract

The growth in the number of computational resources used by high-performance computing (HPC) systems leads to an increase in failure rates. Fault-tolerant techniques will become essential for long-running applications executing in future exascale systems, not only to ensure the completion of their execution in these systems but also to improve their energy consumption. Although the Message Passing Interface (MPI) is the most popular programming model for distributed-memory HPC systems, as of now, it does not provide any fault-tolerant construct for users to handle failures. Thus, the recovery procedure is postponed until the application is aborted and re-spawned. The proposal of the User Level Failure Mitigation (ULFM) interface in the MPI forum provides new opportunities in this field, enabling the implementation of resilient MPI applications, system runtimes, and programming language constructs able to detect and react to failures without aborting their execution. This paper presents a global overview of the resilience interfaces provided by the ULFM specification, covers archetypal usage patterns and building blocks, and surveys the wide variety of application-driven solutions that have exploited them in recent years. The large and varied number of approaches in the literature proves that ULFM provides the necessary flexibility to implement efficient faulttolerant MPI applications. All the proposed solutions are based on application-driven recovery mechanisms, which allows reducing the overhead and obtaining the required level of efficiency needed in the future exascale platforms.

Resilience and fault tolerance in high-performance computing for numerical weather and climate prediction

“…Although not yet part of the standard, the proposed ULFM extensions have already been applied in a number of studies (see, e.g. Ali et al, 2014; Ashraf et al, 2018; Bland et al, 2013; Cantwell and Nielsen, 2019; Engwer et al, 2018; Fagg and Dongarra, 2000; Gamell et al, 2017a, 2017b; Losada et al, 2020; Teranishi and Heroux, 2014).…”

“…User-level failure mitigation (ULFM) extensions to the MPI standard provide an application-driven mechanism to detect and recover communication channels in the event of a hard fault leading to the loss of one or more processes. These extensions are currently under consideration for inclusion in the MPI 4.x standard (Losada et al, 2020). In prior versions of MPI, communication routines would either trigger an immediate abortion of the program or return control to the application to support a more controlled termination.…”

A Robust Distributed Clustering of Large Data Sets on a Grid of Commodity Machines

“…However, MPI was found efficient only when the size of the data set is small or moderate [17]. Moreover, MPI does not provide any fault-tolerant constructs for users to handle failures [18]. Thus, the recovery procedure is activated only when the application is aborted.…”

Predictive Reliability and Fault Management in Exascale Systems

“…Several programming models include now resilience support. In Reference [ 82 ], authors provide a detailed analysis of the resilience features of the different programing languages, grouped by paradigm: message passing (e.g., MPI-ULFM [ 118 ]), partitioned global address space (e.g., UPC++ [ 12 ]), asynchronous partitioned global address space (e.g., X10 [ 32 ]), actor (e.g., Erlang [ 174 ]), dataflow (e.g., Legion [ 13 ]). Table 2 summarizes the comparison developed in Reference [ 82 ].…”