Online Anomaly Prediction for Robust Cluster Systems

Abstract

Abstract-In this paper, we present a stream-based mining algorithm for online anomaly prediction. Many real-world applications such as data stream analysis requires continuous cluster operation. Unfortunately, today's large-scale cluster systems are still vulnerable to various software and hardware problems. System administrators are often overwhelmed by the tasks of correcting various system anomalies such as processing bottlenecks (i.e., full stream buffers), resource hot spots, and service level objective (SLO) violations. Our anomaly prediction scheme raises early alerts for impending system anomalies and suggests possible anomaly causes. Specifically, we employ Bayesian classification methods to capture different anomaly symptoms and infer anomaly causes. Markov models are introduced to capture the changing patterns of different measurement metrics. More importantly, our scheme combines Markov models and Bayesian classification methods to predict when a system anomaly will appear in the foreseeable future and what are the possible anomaly causes. To the best of our knowledge, our work provides the first stream-based mining algorithm for predicting system anomalies. We have implemented our approach within the IBM System S distributed stream processing cluster, and conducted case study experiments using fully implemented distributed data analysis applications processing real application workloads. Our experiments show that our approach efficiently predicts and diagnoses several bottleneck anomalies with high accuracy while imposing low overhead to the cluster system.

Message-Oriented Middleware with QoS Awareness

“…We provide a path computation algorithm that takes into account such failure probabilities towards choosing the most resilient combination of parallel paths. The failure probabilities of brokers and links are assumed to be known in advance, while our algorithm can accommodate various definitions of resiliency such as [11] or using historic information. For example, the percentage of time that a broker is available in a specific operational period of time can be extracted from traces such as the all-pairs-pings service.…”

Adaptive system anomaly prediction for large-scale hosting infrastructures

“…To achieve generality, the ALERT system is implemented based on standard Linux APIs, which allows us to port the ALERT system to different hosting infrastructure easily. We collect about 20 metrics on each host in IBM System S [23,24] and about 66 metrics on each host in PlanetLab [4]. Table 1 lists a subset of key metrics collected by ALERT on System S and PlanetLab.…”

“…For example, we collect about 20 metrics on each host in IBM System S [23,24] and about 66 metrics on each host in PlanetLab [4]. The monitoring sensor periodically samples each metric value at a certain rate (e.g., one sample every 10 seconds) to form a measurement stream.…”

“…In contrast, the proactive approach offers better system reliability but can incur prohibitive overhead. This motivates us to explore a new predictive anomaly management approach [23,24] that can raise advance anomaly alerts to trigger proper anomaly correction in a just-in-time fashion.…”

On Predictability of System Anomalies in Real World

“…Although previous work (e.g., [10], [11], [12]) has addressed the anomaly detection problem, anomaly prediction needs to capture pre-anomaly symptoms to raise advance anomaly alert before the anomaly happens. In [13], we presented the initial design of our online anomaly prediction scheme. However, one big question is whether real system anomalies do exhibit certain predictability and whether our anomaly prediction scheme can efficiently capture the predictability.…”