Improving cluster availability using workstation validation

Abstract

We demonstrate a framework for improving the availability of cluster based Internet services. Our approach models Internet services as a collection of interconnected components, each possessing well defined interfaces and failure semantics. Such a decomposition allows designers to engineer high availability based on an understanding of the interconnections and isolated fault behavior of each component, as opposed to ad-hoc methods. In this work, we focus on using the entire commodity workstation as a component because it possesses natural, fault-isolated interfaces. We define a failure event as a reboot because not only is a workstation unavailable during a reboot, but also because reboots are symptomatic of a larger class of failures, such as configuration and operator errors. Our observations of 3 distinct clusters show that the time between reboots is best modeled by a Weibull distribution with shape parameters of less than 1, implying that a workstation becomes more reliable the longer it has been operating. Leveraging this observed property, we design an allocation strategy which withholds recently rebooted workstations from active service, validating their stability before allowing them to return to service. We show via simulation that this policy leads to a 70-30 rule-of-thumb: For a constant utilization, approximately 70% of the workstation failures can be masked from end clients with 30% extra capacity added to the cluster, provided reboots are not strongly correlated. We also found our technique is most sensitive to the burstiness of reboots as opposed to absolute lengths of workstation uptimes.

On the impact of process replication on executions of large-scale parallel applications with coordinated checkpointing

“…Although our results for two particular production workloads show that Daly's approximation leads to reasonably good results in the no-replication case (see the g = 1 curves in Fig. 5), there is evidence that, in general, failure distributions are well approximated by Weibull distributions [17][18][19]21,20], while not at all by exponential distributions. Most recently, in [21], the authors show that failures observed on a production cluster, over a cumulative 42-month time period, are modeled well by a Weibull distribution with shape parameter k < 0.5.…”

“…Exponential failures are often assumed due to their convenient memoryless property, i.e., the fact that the time to the next failure does not depend on when the last failure has occurred [16]. But the non-memoryless Weibull distribution is recognized as a more realistic model [17][18][19][20][21]. The work in this paper relates to CRR in the sense that we study a replication mechanism that is complementary to checkpointing.…”

An optimistic checkpoint mechanism based on job characteristics and resource availability for dynamic grids

“…More attention has been paid to modeling the characteristics of resource availability and many researches [5,[8][9][10][21][22][23] show that strong temporal and spatial correlations of failure events and resource failures follow the Weibull, Hyperexponential and Pareto distributions with different parameters rather than a Poisson distribution. Oliner et al [4] and Zhang et al [5] evaluate three application-level periodic checkpoint heuristics, checkpointing all jobs, long jobs and big jobs, in large-scale cluster system using temporal or spatial information of resource availability.…”

“…For example, the job whose running time exceeds one hour(it can be pre-specified) and whose running success ratio is less than 90%(it can be pre-specified) can be defined as the long job. RT ij denotes the running time of job i on resource node j and can be obtained according to (9).…”

“…[6] assumes that the resource failures follow a Weibull distribution with shape parameter=1.4. However, many literatures [5,[8][9][10] show that the resource failures follow a Weibull distribution with shape parameter less than 1, implying that the resource becomes more reliable the longer it has been operating, in other words, the resource failure rate (λ(t)) becomes lower the longer it has been operating. Thus, in terms of equation (1), the checkpoint frequency would become smaller the longer the system has been operating.…”