File I/O for MPI Applications in Redundant Execution Scenarios

Abstract

As multi-petascale and exa-scale highperformance computing (HPC) systems inevitably have to deal with a number of resilience challenges, such as a significant growth in component count and smaller circuit sizes with lower circuit voltages, redundancy may offer an acceptable level of resilience that traditional fault tolerance techniques, such as checkpoint/restart, do not. Although redundancy in HPC is quite controversial due to the associated cost for redundant components, the constantly increasing number of cores-per-processor is tilting this cost calculation toward a system design where computation, such as for redundancy, is much cheaper and communication, needed for checkpoint/restart, is much more expensive. Recent research and development activities in redundancy for Message Passing Interface (MPI) applications focused on availability/reliability models and replication algorithms. This paper takes a first step toward solving an open research problem associated with running a parallel application redundantly, which is file I/O under redundancy. The approach intercepts file I/O calls made by a redundant application to employ coordination protocols that execute file I/O operations in a redundancy-oblivious fashion when accessing a node-local file system, or in a redundancy-aware fashion when accessing a shared networked file system. A proof-of concept prototype is presented and a number of coordination protocols are described and evaluated. The results show the performance impact for redundantly accessing a shared networked file system, but also demonstrate the capability to regain performance by utilizing MPI communication between replicas and parallel file I/O.

DINO: Divergent Node Cloning for Sustained Redundancy in HPC

“…If an algorithm lacks a simple checking method or invariant, the Checker can be provided through comparison with a checksum over the data that was computed beforehand and stored in a safe region. 2 The Recover method can be supplied through the forward recovery phase in ABFT methods, or simply by restoring a light-weight deduplicated [1] or compressed [17] checkpoint of the data.…”

“…CR and redundant computing already ensure idem-potency since identical state is restored in the former while redundant state exists for the latter, but ABFT methods have to be complemented, e.g., by compiler-driven live variable analysis to capture/restore globals at region boundaries. Existing solutions to I/O idem-potency are required as well [2]. We can then retry a computation if needed, i.e., when no other recovery methods exist (or if the other recovery methods have failed).…”

Detection and Correction of Silent Data Corruption for Large-Scale High-Performance Computing

“…As the primary focus of this work is to investigate redundancy as a means to protect application data, RedMPI only attempts to protect against corruption in data and not application code or instructions. RedMPI does not protect MPI I/O functionality, but orthogonal work [22]…”

Replication for send-deterministic MPI HPC applications

“…Note that checkpointing, recovery are not implemented in the current prototype. Since I/O operations are often used to save intermediate results and implement application-level checkpointing, we plan to integrate application level checkpointing using the solution proposed in [1] to handle IO in a replicated MPI application.…”

“…The study presented in [9] is the first showing that active replication could outperform coordinated checkpointing at scale 1 . In that work, replication is combined with coordinated checkpointing: If each process is replicated, the probably that the application needs to be restarted from a checkpoint, meaning that all replicas of one process have failed, is dramatically reduced compared to a scenario without replication.…”