A Fault Tolerant MPI-IO Implementation using the Expand Parallel File System

Abstract

Parallelism in file systems is obtained by using several independent server nodes supporting one or more secondary storage devices. This approach increases the performance and scalability of the system, but a fault in one single node can stop the whole system. To avoid this problem, data must be stored using some kind of redundant technique, so any data stored in a faulty element can be recovered. Fault tolerance can be provided in I/O systems using replication or RAID based schemes. However, most of the current systems apply the same technique for all files in the system. 1 This paper describes the fault tolerance support provided by Expand, a parallel file system based on standard servers. Expand allows to define different fault-tolerant mechanisms at file level. The evaluation compare the performance of Expand with different configurations with PVFS using the FLASH-I/O benchmark.

Fault tolerant file models for parallel file systems: introducing distribution patterns for every file

“…By using different storage spaces for user data and redundant information, Expand can add and remove fault-tolerant support on a file basis [15].…”

“…In a previous paper in bibliography [15], we presented the implementation and evaluation of the first prototype, with different fault tolerant schemes and using MPI-IO interface. In this paper, we present the design and the initial framework behind the fault tolerant architecture of Expand.…”

“…In Expand, fault tolerance is provided at file level [14,15], because not all files have the same requirements of fault tolerance degree and performance. In fact, the number of critical files for a user during a period of time is a small part of all files that belong to him, and this subset of files changes along the time.…”

Handling Persistent States in Process Checkpoint/Restart Mechanisms for HPC Systems

“…In the Expand parallel file system [2], each file can have its own fault tolerance policy (no fault tolerance, replication, parity based redundancy, etc.). This can be seen as similar, however more powerful, to the difference we make between files involved in a checkpoint (which are replicated) and other files (which are not).…”