Improving the Computing Efficiency of HPC Systems Using a Combination of Proactive and Preventive Checkpointing

Bouguerra, Mohamed Slim; Gainaru, Ana; Bautista-Gomez, Leonardo; Cappello, Franck; Matsuoka, Satoshi; Maruyama, Naoya

doi:10.1109/ipdps.2013.74

Cited by 42 publications

(32 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other advances concern the combination of application-level checkpointing and failure prediction [10]. An important question is how to run the failure predictor on large infrastructures.…”

Section: Failure Predictionmentioning

confidence: 99%

“…This approach faces two difficulties: (1) local failure prediction will impose an overhead on the application running on the node, and (2) local failure prediction is less accurate than global failure prediction because the failure predictors have only a local view. These two difficulties are explained in [10]. Another important question is how to compute the optimal interval of preventive checkpoints when a proportion of the failures are predicted [10].…”

Section: Failure Predictionmentioning

confidence: 99%

“…These two difficulties are explained in [10]. Another important question is how to compute the optimal interval of preventive checkpoints when a proportion of the failures are predicted [10]. In particular, many formulations of the optimal checkpoint interval problem consider that failures follow an exponential distribution of interarrival times.…”

Section: Failure Predictionmentioning

confidence: 99%

See 2 more Smart Citations

Toward Exascale Resilience: 2014 update

Cappello

Geist

Gropp

et al. 2014

JSFI

View full text Add to dashboard Cite

Resilience is a major roadblock for HPC executions on future exascale systems. These systems will typically gather millions of CPU cores running up to a billion threads. Projections from current large systems and technology evolution predict errors will happen in exascale systems many times per day. These errors will propagate and generate various kinds of malfunctions, from simple process crashes to result corruptions.The past five years have seen extraordinary technical progress in many domains related to exascale resilience. Several technical options, initially considered inapplicable or unrealistic in the HPC context, have demonstrated surprising successes. Despite this progress, the exascale resilience problem is not solved, and the community is still facing the difficult challenge of ensuring that exascale applications complete and generate correct results while running on unstable systems. Since 2009, many workshops, studies, and reports have improved the definition of the resilience problem and provided refined recommendations. Some projections made during the previous decades and some priorities established from these projections need to be revised. This paper surveys what the community has learned in the past five years and summarizes the research problems still considered critical by the HPC community.

show abstract

“…Other advances concern the combination of application-level checkpointing and failure prediction [10]. An important question is how to run the failure predictor on large infrastructures.…”

Section: Failure Predictionmentioning

confidence: 99%

Section: Failure Predictionmentioning

confidence: 99%

Section: Failure Predictionmentioning

confidence: 99%

See 1 more Smart Citation

Toward Exascale Resilience: 2014 update

Cappello

Geist

Gropp

et al. 2014

JSFI

View full text Add to dashboard Cite

show abstract

“…If multiple applications run concurrently, a dataaware compression scheme [29] was proposed to improve the overall checkpointing efficiency. Recent study [30] shows that combining failure detection and proactive checkpointing could improve 30% efficiency compared to classical periodical checkpointing. Thus data compression has the potential to be combined with failure detection and proactive checkpointing to further improve the system efficiency.…”

Section: Related Workmentioning

confidence: 99%

Virtual chunks: On supporting random accesses to scientific data in compressible storage systems

Zhao

Yin

Qiao

et al. 2014

2014 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

Abstract-Data compression could ameliorate the I/O pressure of scientific applications on high-performance computing systems. Unfortunately, the conventional wisdom of naively applying data compression to the file or block brings the dilemma between efficient random accesses and high compression ratios. Filelevel compression can barely support efficient random accesses to the compressed data: any retrieval request need trigger the decompression from the beginning of the compressed file. Block-level compression provides flexible random accesses to the compressed data, but introduces extra overhead when applying the compressor to each every block that results in a degraded overall compression ratio. This paper introduces a concept called virtual chunks aiming to support efficient random accesses to the compressed scientific data without sacrificing its compression ratio. In essence, virtual chunks are logical blocks identified by appended references without breaking the physical continuity of the file content. These additional references allow the decompression to start from an arbitrary position (efficient random access), and retain the file's physical entirety to achieve high compression ratio on par with file-level compression. One potential concern of virtual chunks lies on its space overhead (from the additional references) that degrades the compression ratio, but our analytic study and experimental results demonstrate that such overhead is negligible. We have implemented virtual chunks in two forms: a middleware to the GPFS parallel file system, and a module in the FusionFS distributed file system. Large-scale evaluations on up to 1,024 cores showed that virtual chunks could help improve the I/O throughput by 2X speedup.

show abstract

“…Event analysis and classification in large-scale system have been the subject of many studies, and research has shown that they can lead to good prediction results [8], [9]. One of the major challenges in this endeavor is the impact of false positives.…”

Section: B Introspective Systemsmentioning

confidence: 99%

Monitoring strategies for scalable dynamic checkpointing

Perarnau

Bautista-Gomez

2016

2016 Seventh International Green and Sustainable Computing Conference (IGSC)

View full text Add to dashboard Cite

Abstract-Resilience is an important challenge for extremescale supercomputers. Failures in current supercomputers are assumed to be uniformly distributed in time. However, recent studies show that failures in high-performance computing systems are partially correlated in time, generating periods of higher failure density. The detection of those periods is important in order to adjust the system to new conditions. In this paper we present a monitoring system that listens to hardware events across computing nodes and forwards important events to the fault tolerance runtime so it can react to those regime changes. Our evaluation at scale shows several aspects of this dynamic checkpointing scheme, critical to understanding its applicability on production systems, as well as to identifying possible avenues for future improvements. In particular, we evaluate the ability of our system to monitor as many types of events as possible, measure their importance, and forward them to the resilience runtime.

show abstract

Improving the Computing Efficiency of HPC Systems Using a Combination of Proactive and Preventive Checkpointing

Cited by 42 publications

References 28 publications

Toward Exascale Resilience: 2014 update

Toward Exascale Resilience: 2014 update

Virtual chunks: On supporting random accesses to scientific data in compressible storage systems

Monitoring strategies for scalable dynamic checkpointing

Contact Info

Product

Resources

About