Versioned Distributed Arrays for Resilience in Scientific Applications: Global View Resilience

Chien, Andrew A.; Balaji, Pavan; Beckman, Peter H.; Dun, Nan; Fang, Aiman; Fujita, Hideaki; Iskra, Kamil; Rubenstein, Zachary A.; Zheng, Ziming; Schreiber, Robert; Hammond, Jeff R.; Dinan, James; Laguna, Ignacio; Richards, David F.; Dubey, Anshu; Straalen, Brian Van; Hoemmen, Mark Frederick; Heroux, Michael A.; Teranishi, Keita; Siegel, A.

doi:10.1016/j.procs.2015.05.187

Cited by 25 publications

(18 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Checkpointing and selection of checkpoints are important for ensuring end-user productivity in case of failures and recovery. 5,49,50 In the proposed approach, violation of the health rules is based on the assumption that the silent errors are almost instantaneous and affect the simulations immediately so that the rule violation is trigged within a few steps of the simulations. If such an event occurs, a healthy or error-free checkpoint is selected from a state that is considered healthy, based on the defined rules.…”

Section: Selection Of Checkpoint Considerationsmentioning

confidence: 99%

Application health monitoring for extreme‐scale resiliency using cooperative fault management

Agarwal

Naughton

Park

et al. 2019

Concurrency and Computation

View full text Add to dashboard Cite

Summary Resiliency is and will be a critical factor in determining scientific productivity on current and exascale supercomputers, and beyond. Applications oblivious to and incapable of handling transient soft and hard errors could waste supercomputing resources or, worse, yield misleading scientific insights. We introduce a novel application‐driven silent error detection and recovery strategy based on application health monitoring. Our methodology uses application output that follows known patterns, as indicators of an application's health and knowledge that violation of these patterns could be indication of faults. Information from system monitors that report hardware and software health status is used to corroborate faults. Collectively, this information is used by a fault coordinator agent to take preventive and corrective measures by applying computational steering to an application between checkpoints. This cooperative fault management system uses the Fault Tolerance Backplane as a communication channel. The benefits of this framework are demonstrated with two real application case studies, molecular dynamics, and quantum chemistry simulations, on scalable clusters with simulated memory and I/O corruptions. The developed approach is general and can be easily applied to other applications.

show abstract

Section: Selection Of Checkpoint Considerationsmentioning

confidence: 99%

Application health monitoring for extreme‐scale resiliency using cooperative fault management

Agarwal

Naughton

Park

et al. 2019

Concurrency and Computation

View full text Add to dashboard Cite

show abstract

“…Ali et al [27] use redundant communication to continuously update shadow data structures. Other user-level approaches target arrays [28] as well as the MapReduce [29] and hierarchical master/worker patterns [30]. Task scheduling in grids and clouds differs from our work in coarser-grained tasks and centralization (e.g.…”

Section: Related Workmentioning

confidence: 99%

Fault Tolerance for Lifeline-Based Global Load Balancing

Fohry¹,

Bungart²,

Plock³

2017

JSEA

View full text Add to dashboard Cite

Fault tolerance has become an important issue in parallel computing. It is often addressed at system level, but application-level approaches receive increasing attention. We consider a parallel programming pattern, the task pool, and provide a fault-tolerant implementation in a library. Specifically, our work refers to lifeline-based global load balancing, which is an advanced task pool variant that is implemented in the GLB framework of the parallel programming language X10. The variant considers side effect-free tasks whose results are combined into a final result by reduction. Our algorithm is able to recover from multiple fail-stop failures. If recovery is not possible, it halts with an error message. In the algorithm, each worker regularly saves its local task pool contents in the main memory of a backup partner. Backups are updated for steals. After failures, the backup partner takes over saved copies and collects others. In case of multiple failures, invocations of the restore protocol are nested. We have implemented the algorithm by extending the source code of the GLB library. In performance measurements on up to 256 places, we observed an overhead between 0.5% and 30%. The particular value depends on the application's steal rate and task pool size. Sources of performance overhead have been further analyzed with a logging component.

show abstract

“…GVR has been used to demonstrate flexible multi-version rollback, forward error correction, and other creative recovery schemes [5,6]. Demonstrations include high-error rates, and results show modest runtime cost (< 1%) and programming effort in fullscale molecular dynamics, Monte Carlo, adaptive mesh, and indirect linear solver applications [7,8].…”

Section: Global View Resilience (Gvr)mentioning

confidence: 99%

“…-Because latent ("silent") errors are complex to identify, the detector is computationally expensive. 7 The interval between two consecutive error detections bounds the error latency. Given the error location and timing, three steps are performed to correct the state of corrupted data.…”

Section: Algorithm-based Focused Recovery (Abfr)mentioning

confidence: 99%

Resilient N-Body Tree Computations with Algorithm-Based Focused Recovery: Model and Performance Analysis

Cavelan

Fang

Chien

et al. 2017

Lecture Notes in Computer Science

View full text Add to dashboard Cite

This paper presents a model and performance study for Algorithm-Based Focused Recovery (ABFR) applied to N-body computations, subject to latent errors. We make a detailed comparison with the classical Checkpoint/Restart (CR) approach. While the model applies to general frameworks, the performance study is limited to perfect binary trees, due to the inherent difficulty of the analysis. With ABFR, the crucial parameter is the detection interval, which bounds the error latency. We show that the detection interval has a dramatic impact on the overhead, and that optimally choosing its value leads to significant gains over the CR approach.

show abstract

Versioned Distributed Arrays for Resilience in Scientific Applications: Global View Resilience

Cited by 25 publications

References 8 publications

Application health monitoring for extreme‐scale resiliency using cooperative fault management

Application health monitoring for extreme‐scale resiliency using cooperative fault management

Fault Tolerance for Lifeline-Based Global Load Balancing

Resilient N-Body Tree Computations with Algorithm-Based Focused Recovery: Model and Performance Analysis

Contact Info

Product

Resources

About