Predicting failures in multi-tier distributed systems

Abstract

Many applications are implemented as multi-tier software systems, and are executed on distributed infrastructures, like cloud infrastructures, to benefit from the cost reduction that derives from dynamically allocating resources ondemand. In these systems, failures are becoming the norm rather than the exception, and predicting their occurrence, as well as locating the responsible faults, are essential enablers of preventive and corrective actions that can mitigate the impact of failures, and significantly improve the dependability of the systems. Current failure prediction approaches suffer either from false positives or limited accuracy, and do not produce enough information to effectively locate the responsible faults.In this paper, we present PreMiSE, a lightweight and precise approach to predict failures and locate the corresponding faults in multi-tier distributed systems. PreMiSE blends anomaly-based and signature-based techniques to identify multi-tier failures that impact on performance indicators, with high precision and low false positive rate. The experimental results that we obtained on a Cloud-based IP Multimedia Subsystem indicate that PreMiSE can indeed predict and locate possible failure occurrences with high precision and low overhead. Figure 1: The overall flow of PreMiSE online activities to predict failures ing failures that impact on performance indicators. As illustrated in Figure 1, PreMiSE -monitors the status of the system by collecting (a large set of) performance indicators from the system nodes, for instance CPU utilization for each CPU in the system, that we refer to as Key Performance Indicators (KPIs) (KPI monitoring in the figure), -identifies deviations from normal behaviors by pinpointing anomalous KPIs with anomaly-based techniques (Anomaly detection in the figure), -identifies incoming failures by identifying symptomatic anomalous KPI sets with signature-based techniques.(Signature-based failure prediction in the figure).

Cloud Failure Prediction with Hierarchical Temporal Memory: An Empirical Assessment

“…Compared to studies where failures are used in the training phase [10], [13], HTM achieved slightly lower effectiveness. Although results obtained in different experiments are not directly comparable, a gap was expected.…”

“…Note that failure-free executions are not free of anomalies, since services may operate in an unpredictable way, despite not generating any failure. To generate the traffic for the running system, we generated a number of users and calls according to weekly and daily patterns (e.g., more users on workdays, fewer users at night, pick time at 9am and 7pm), including a certain degree of randomness, as done in similar studies that used Clearwater as subject system [10].…”

“…We study the effectiveness of HTM in supporting cloud failure prediction by using a KPI-driven online failure prediction approach, which analyzes the KPIs to detect the patterns of anomalous behaviors that are likely to result in failures at a later time. KPI-driven online failure prediction is implemented by many techniques [3], [10], [13] and can be considered as the most used approach to failure prediction in the Cloud.…”

“…For example, Lin et al [3] used an ensemble of supervised machine learning models to predict failures in cloud systems. Mariani et al [10] combined anomalybased and signature-based techniques for predicting failures in multi-tier distributed systems. Supervised learning is often impractical, since it needs generating many sample failures, while changes to the system require retraining the models.…”

A Taxonomy of Techniques for SLO Failure Prediction in Software Systems

“…This is the improved version of their offline technique already introduced in Section 5.3.1. Lastly, Mariani et al [80] aim at predicting failures in distributed multi-tier environments by employing a two-stage approach, consisting of anomaly detection in the first stage, and a signature-based classification technique in the second stage.…”