Lightweight storage and overlay networks for fault tolerance.

Abstract

The next generation of capability-class, massively parallel processing (MPP) systems is expected to have hundreds of thousands to millions of processors, In such environments, it is critical to have fault-tolerance mechanisms, including checkpoint/restart, that scale with the size of applications and the percentage of the system on which the applications execute. For application-driven, periodic checkpoint operations, the state-of-the-art does not provide a scalable solution. For example, on today's massive-scale systems that execute applications which consume most of the memory of the employed compute nodes, checkpoint operations generate I/O that consumes nearly 80% of the total I/O usage. Motivated by this observation, this project aims to improve I/O performance for application-directed checkpoints through the use of lightweight storage architectures and overlay networks. Lightweight storage provide direct access to underlying storage devices. Overlay networks provide caching and processing capabilities in the compute-node fabric. The combination has potential to signifcantly reduce I/O overhead for large-scale applications. This report describes our combined efforts to model and understand overheads for application-directed checkpoints, as well as implementation and performance analysis of a checkpoint service that uses available compute nodes as a network cache for checkpoint operations.3

Using Probabilistic Characterization to Reduce Runtime Faults in HPC Systems

“…They further make the case that it is likely that aggregate computational power for the foreseeable future will come from just such increases. Their exploration of current and alternative checkpointing techniques and associated pros and cons parallels those of others such as Oldfield [3]. All such work seems to draw the conclusion that current fault tolerance techniques, namely checkpoint and restart, in their current form coupled with the seemingly fixed relationship between number of CPU sockets and MTTI imply that run time efficiency will continue to fall as computational platforms become larger.…”

Data co-processing for extreme scale analysis level II ASC milestone (4745).

“…For example, we developed services for staging checkpoint data [23,24,31], HPC database integration [30], interactive visualization [25], network traffic analysis, and most recently CTH in transit analysis [22]. A recent paper describes these services in detail [17].…”

Report of experiments and evidence for ASC L2 milestone 4467 : demonstration of a legacy application's path to exascale.

“…One approach to both reduce the burden on the file system and improve "effective" I/O rates for the application is to employ the use of "data services" [1,85,97,100,139]. Simply put, a data service is a separate (possibly parallel) application that performs operations on behalf of an actively running scientific application.…”

“…On current capability-class HPC systems, services execute on compute nodes or service nodes and provide the application the ability to "offload" operations that present scalability challenges for the scientific code. One commonly used example for data services is data staging, or caching data between the application and the storage system [100,101,118]. Section 7.4 describes such a service.…”