Performance Implications of Failures in Large-Scale Cluster Scheduling

Abstract

Abstract. As we continue to evolve into large-scale parallel systems, many of them employing hundreds of computing engines to take on mission-critical roles, it is crucial to design those systems anticipating and accommodating the occurrence of failures. Failures become a commonplace feature of such large-scale systems, and one cannot continue to treat them as an exception. Despite the current and increasing importance of failures in these systems, our understanding of the performance impact of these critical issues on parallel computing environments is extremely limited. In this paper we develop a general failure modeling framework based on recent results from large-scale clusters and then we exploit this framework to conduct a detailed performance analysis of the impact of failures on system performance for a wide range of scheduling policies. Our results demonstrate that such failures can have a significant impact on the mean job response time and mean job slowdown under existing scheduling policies that ignore failures. We therefore investigate different scheduling mechanisms and policies to address these performance issues. Our results show that periodic checkpointing of jobs seems to do little to ease this problem. On the other hand, we demonstrate that information about the spatial and temporal correlation of failure occurrences can be very useful in designing a scheduling (job allocation) strategy to enhance system performance, with the former providing the greatest benefits.

A Model for Space-Correlated Failures in Large-Scale Distributed Systems

“…A bursty arrival breaks an important assumption made by numerous fault tolerant algorithms [8,20,15], that of independent and identical distribution of failures among the components of the system. However, few studies [19,4,11] investigate the bursty arrival of failures for distributed systems.…”

“…The importance of spacecorrelated failures has been repeatedly noted: the availability of a distributed system may be overestimated by an order of magnitude when as few as 10% of the failures are correlated [19], and a halving of the work loss may be achieved when taking into account space-correlated failures [20].…”

Providing Fault-Tolerance in Unreliable Grid Systems Through Adaptive Checkpointing and Replication

“…Studies of real systems, however, show that failures are correlated temporally and spatially, are not identically distributed. Furthermore, the behavior of checkpointing schemes under these realistic failure distributions does not follow the behavior predicted by standard checkpointing models [2,6].…”

“…The latter possess a varying failure and restore behavior, which is modelled to mimic reality as much as possible. As outlined by Zhang et al [6], failures in large-scale distributed systems are mostly correlated and tend to occur in bursts. Besides, there are strong spatial correlations between failures and nodes, where a small fraction of the nodes incur most of the failures.…”

IMCLA: Performance Evaluation of Integrated Multilevel Checkpointing Algorithms using Checkpointing Efficiency

“…Pausing, resuming, and migration VMs [22], [23] are powerful mechanisms to manage failures in such situations. The checkpointing and rollback recovery technique [24] has been widely used in distributed systems.…”