ABARIS: An Adaptable Fault Detection/Recovery Component Framework for MPIs

Abstract

Long-running MPI applications on clusters and grids that are prone to node and network failures, motivates the use of fault tolerant MPI implementations. However, previous fault tolerant MPIs lack the ability to allow the user to easily choose appropriate fault recovery strategies according to the execution environment, independent of the application codes-rather, the user often had to hard-code restoration strateties in accordance to diverse sets of fault patterns, which could be numerous: for instance, if the fault is transient to a particular process, we merely have to restart the process on the same computing node; on the other hand, if the fault is due to repetitive hardware unreliability, we must migrate the process to a new node in its recovery. ABARIS is our new Fault/Recovery model aware component framework for MPI, where users can customize MPI fault detection and recovery algorithms according to their application and execution environmental requirements by merely selecting appropriate fault/recovery components, independent of the application code. Currently, the ARA-BIS framework prototype is implemented on top of MPICH-P4MPD. Preliminary evaluation of the prototype using NPB on our MPI fault simulator demonstrates that overhead compared to the original MPICH-P4MPD is almost negligible (less than 1%) under normal execution, and when faults occur, appropriate selections and pairings of fault model and recovery method components for corresponding to the execution environment is significant to the overall execution time.

Assessing HPC Failure Detectors for MPI Jobs

“…This paper focuses on the former techniques. It builds on recently developed techniques such as checkpointing (with restarts or rollbacks) or redundant computing in highperformance computing [3,6,13,21,34] or API extensions for checkpointing [15,24]. A common challenge of transparent resiliency lies in the detection of faults, which is also a requirement for fault-awareness as proposed in MPI-3 [1].…”

Open Issues in MPI Implementation

“…MPI implementations must improve their ability to handle failures, such as broken connections and dead processes, to the extent possible. A number of research efforts in faulttolerant MPI implementation exist [2,7,13]. However, production MPI implementations need to improve their support for fault tolerance.…”

Reliability of Grid Computing Systems

“…This system could initiate different recovery actions designed to remedy different fault types. Jitsumoto and colleagues [44] also developed a detector that differentiated between hardware, process, and transmission faults. Here, users were allowed to pre-select a recovery procedure to be invoked in response to occurrence of a particular fault type.…”

“…If the host processor has not failed, temporal redundancy can be used to roll back and restart the process on the same platform. As in other systems, this method is widely used in grids [16,44,47]. However, if the host has failed, the process may be migrated, or transferred, to a different execution environment.…”

“…Here, the results also indicated that blocking reduced efficiency, while a non-blocking approach appeared to scale well but suffered from implementation issues. Coordinated checkpointing has been implemented in MPI, in LAM/MPI [65] as well as in [44,62,66,67]. Yeom et al [66,67] proposed a faulttolerant version of MPI, MPICH-GF, for grid systems that employed coordinated checkpoints with blocking.…”