Reducing Waste in Extreme Scale Systems through Introspective Analysis

Bautista-Gomez, Leonardo; Gainaru, Ana; Perarnau, Swann; Tiwari, Devesh; Gupta, Saurabh; Engelmann, Christian; Cappello, Franck; Snir, Marc

doi:10.1109/ipdps.2016.100

Cited by 26 publications

(31 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While we demonstrated previously that such scheme could result in satisfying results [7], one could imagine more complex identifiers using real-time information on the system to adapt its detection.…”

Section: B Identifying Regime Changesmentioning

confidence: 90%

“…In a previous work [7], we show how to leverage failure correlation to reduce wasted time. Although that work included a preliminary analysis of the different types of event and a simple monitoring tool, the paper focuses more on giving a picture of the whole structure and not on monitoring.…”

Section: Related Workmentioning

confidence: 99%

“…Although current systems are unable to recognize in real time when the system is functioning in degraded mode, we recently developed a monitoring system and the accompanying dynamic checkpointing runtime to take advantage of this information [7]. Our system design is straightforward: daemons running on all compute nodes monitor as many types of events as possible, use operational knowledge to determine whether those events are the sign of degraded regimes, and forward the critical events to the checkpointing runtime.…”

Section: Introductionmentioning

confidence: 99%

“…It might also require certain changes in the application's resilience mechanisms to make them more dynamic than current standards. Our previous work [7] shows how such dynamic runtime can be implemented and the theoretical benefits that such a scheme might offer. In this paper we focus more on the monitoring aspects of that system, in order to increase the number of correct regime detections and decrease the application overhead generated by false positives.…”

Section: B Introspective Systemsmentioning

confidence: 99%

See 3 more Smart Citations

Monitoring strategies for scalable dynamic checkpointing

Perarnau

Bautista-Gomez

2016

2016 Seventh International Green and Sustainable Computing Conference (IGSC)

Self Cite

View full text Add to dashboard Cite

Abstract-Resilience is an important challenge for extremescale supercomputers. Failures in current supercomputers are assumed to be uniformly distributed in time. However, recent studies show that failures in high-performance computing systems are partially correlated in time, generating periods of higher failure density. The detection of those periods is important in order to adjust the system to new conditions. In this paper we present a monitoring system that listens to hardware events across computing nodes and forwards important events to the fault tolerance runtime so it can react to those regime changes. Our evaluation at scale shows several aspects of this dynamic checkpointing scheme, critical to understanding its applicability on production systems, as well as to identifying possible avenues for future improvements. In particular, we evaluate the ability of our system to monitor as many types of events as possible, measure their importance, and forward them to the resilience runtime.

show abstract

Section: B Identifying Regime Changesmentioning

confidence: 90%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: B Introspective Systemsmentioning

confidence: 99%

See 2 more Smart Citations

Monitoring strategies for scalable dynamic checkpointing

Perarnau

Bautista-Gomez

2016

2016 Seventh International Green and Sustainable Computing Conference (IGSC)

Self Cite

View full text Add to dashboard Cite

show abstract

“…We base our analysis on publicly available failure logs from LANL [15,14] and Tsubame [23]. We show that a previously proposed approach based on degraded intervals [3] leads to incorrect results, and we propose a new algorithm to detect failure cascades, based on the study of pairs of consecutive IATs. This new algorithm is used for the largest six public logs at our disposal, and we detect cascades in one log for sure, and possibly in a second one.…”

Section: Introductionmentioning

confidence: 99%

Assuming Failure Independence: Are We Right to be Wrong?

Aupy

Robert

Vivien

2017

2017 IEEE International Conference on Cluster Computing (CLUSTER)

View full text Add to dashboard Cite

This report revisits the failure temporal independence hypothesis which is omnipresent in the analysis of resilience methods for HPC. We explain why a previous approach is incorrect, and we propose a new method to detect failure cascades, i.e., series of non-independent consecutive failures. We use this new method to assess whether public archive failure logs contain failure cascades. Then we design and compare several cascadeaware checkpointing algorithms to quantify the maximum gain that could be obtained, and we report extensive simulation results with archive and synthetic failure logs. Altogether, not only are there but a few logs that contain cascades, but we show that the gain that can be achieved from this knowledge is not significant. The conclusion is that we can wrongly, but safely, assume failure independence!

show abstract

FATM: A failure‐aware adaptive fault tolerance model for distributed stream processing systems

Akber

Chen

Jin

2021

Concurrency and Computation

View full text Add to dashboard Cite

Summary Distributed Stream Processing Systems (DSPS) are very popular to process unbounded data streams in real‐time. Low processing latency is a fundamental requirement for DSPS applications to maintain the real‐time response. This requirement of low processing latency for DSPS is badly affected due to inevitable failures in computing systems. Generally, DSPS grapple with these inevitable failures by triggering periodic checkpoints. The periodic checkpoints pessimistically persist the application state so that the execution may be resumed after the failure. These periodic checkpoints incur high overheads due to the high frequency of checkpoints triggering, which increases the overall execution time. On the other hand, failure occurrences in real‐world systems are not periodic. This sharp contrast between the periodic checkpoints and failure distributions in the real‐world systems makes the periodic checkpoints inefficient. We propose a failure‐aware adaptive fault tolerance model called FATM which triggers the checkpoints inline with the underlying failure rate. Further, we design a model for utility factor and checkpoint overheads to evaluate the performance of fault tolerance models for DSPS. We implement the FATM atop Apache Flink and perform a series of experiments. To validate the effectiveness of FATM, experiment results are compared with the existing checkpoint‐based models of DSPS. The results show that the FATM significantly reduces the checkpoint frequency, increases the utility factor, and reduces the checkpoint overheads by 28%.

show abstract

Reducing Waste in Extreme Scale Systems through Introspective Analysis

Cited by 26 publications

References 17 publications

Monitoring strategies for scalable dynamic checkpointing

Monitoring strategies for scalable dynamic checkpointing

Assuming Failure Independence: Are We Right to be Wrong?

FATM: A failure‐aware adaptive fault tolerance model for distributed stream processing systems

Contact Info

Product

Resources

About