CloudPD: Problem determination and diagnosis in shared dynamic clouds

Abstract

Abstract-In this work, we address problem determination in virtualized clouds. We show that high dynamism, resource sharing, frequent reconfiguration, high propensity to faults and automated management introduce significant new challenges towards fault diagnosis in clouds. Towards this, we propose CloudPD, a fault management framework for clouds. CloudPD leverages (i) a canonical representation of the operating environment to quantify the impact of sharing; (ii) an online learning process to tackle dynamism; (iii) a correlation-based performance models for higher detection accuracy; and (iv) an integrated end-to-end feedback loop to synergize with a cloud management ecosystem. Using a prototype implementation with cloud representative batch and transactional workloads like Hadoop, Olio and RUBiS, it is shown that CloudPD detects and diagnoses faults with low false positives (< 16%) and high accuracy of 88%, 83% and 83%, respectively. In an enterprise trace-based case study, CloudPD diagnosed anomalies within 30 seconds and with an accuracy of 77%, demonstrating its effectiveness in real-life operations.

Performance Anomaly Detection and Bottleneck Identification

“…New performance observations that fail to match (within some acceptable confidence levels) existing models are flagged anomalous and system administrators alerted accordingly [39]. However, many solutions employ more complicated techniques (such as statistical and learning methods) while following one or more strategies to achieve some detection goals.…”

“…These VMs concurrently share and compete for virtualized resources (such as CPU and memory) and non virtualized resources (such as network and caches). Such an tight execution environment has been shown to account for 40% in performance degradation in some applications [39]. This makes it essential for PADBI techniques to be aware of prevailing execution contexts.…”

“…Similarly there are systems tailored to detecting and resolving workload related anomalies [ [36]; [41]]. Perhaps due to scale and the special requirements imposed by the cloud, advanced machine learning techniques have found extensive use in bottleneck and anomaly detection research [ [62]; [55]; [40]; [39]]. Even though existing research contribution is dominated by reactive solutions, there is increasing shift towards proactive approach.…”

“…A cloud infrastructure is typically characterized by a pool of heterogeneous hardware and software resources that are shared by many application services with disparate performance objectives [ [135]; [136]]. The resulting resource contention and performance interference caused by resource sharing have significant impact on the performance of cloud services and systems [ [39]; [122]]. …”

Program Aging and Service Crash

“…Service performance is portrayed by reaction time, profitability, accessibility, and security. Quality of Service (QoS) in cloud shows level of performance and dependability, regardless of the fact that attributes of quality of service got consideration before development of cloud, performance, homogeneity as well as principles [58].…”