The Lam/Mpi Checkpoint/Restart Framework: System-Initiated Checkpointing

Abstract

As high performance clusters continue to grow in size and popularity, issues of fault tolerance and reliability are becoming limiting factors on application scalability. To address these issues, we present the design and implementation of a system for providing coordinated checkpointing and rollback recovery for MPI-based parallel applications. Our approach integrates the Berkeley Lab BLCR kernel-level process checkpoint system with the LAM implementation of MPI through a defined checkpoint/restart interface. Checkpointing is transparent to the application, allowing the system to be used for cluster maintenance and scheduling reasons as well as for fault tolerance. Experimental results show negligible communication performance impact due to the incorporation of the checkpoint support capabilities into LAM/MPI.

Checkpointing and Migration of Communication Channels in Heterogeneous Grid Environments

“…Overall there is only one other project working on checkpointing in heterogeneous grid environments while there are different projects implementing checkpointing for MPI applications [11]. However, there are many publications proposing sophisticated checkpointing protocols but that are not related to heterogeneity challenges addressed by this paper.…”

Proactive process-level live migration and back migration in HPC environments

“…Our work enhances LAM/MPI and BLCR [43], [15], [41], which previously was restricted to reactive FT, to a proactive live migration scheme. LAM/MPI+BLCR originally required a complete system restart for roll-back to the last checkpoint upon failure, but a number of approaches have been designed to allow (a) selected checkpoint images to be restarted on new nodes [6], (b) node and head-node failure [47], and (c) a job-pause mechanism that supports migration without restart [51].…”

“…In the context of HPC, many MPI implementations have been retrofitted with or design for FT, ranging from automatic methods (checkpoint-based or log-based) [44], [41], [5] to nonautomated approaches [3], [17].…”

“…BLCR is an open source, systemlevel C/R implementation integrated with LAM/MPI via a callback function. The original LAM/MPI+BLCR combination [41] only provides reactive FT and requires a complete job restart from the last checkpoint including job resubmission in case of a node failure. Recent work enhances this capability with a job pause/continue mechanism that keeps an MPI job alive while a failed node is replaced by a spare node [51].…”

Load Balancing in the Bulk-Synchronous-Parallel Setting using Process Migrations

“…They use an MPI checkpointing system for the LAM MPI implementation [23] based on Berkeley Lab's Linux Checkpoint/Restart [14], a kernel level checkpointing system. As MPI stores the location of the processes, they have to modify the checkpoints before restarting.…”