Efficient incident identification from multi-dimensional issue reports via meta-heuristic search

Gu, Jiazhen; Luo, Chuan; Qin, Si; Qiao, Bo; Lin, Qingwei; Zhang, Hongyu; Li, Ze; Dang, Yingnong; Wu, Wei; Zhou, Yangfan; Chintalapati, Murali; Zhang, Dongmei

doi:10.1145/3368089.3409741

Cited by 22 publications

(3 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Given the dependency among dimensions, HALO [18] formed a hierarchy structure of dimensions instead of the dimension tree based on conditional entropy and searched root causes by walking in this structure. MID [5] improved the efficiency of iDice by an evolution-based search framework. HALO and MID are specially designed for the cloud system.…”

Section: Root Cause Analysismentioning

confidence: 99%

CMMD: Cross-Metric Multi-Dimensional Root Cause Analysis

Yan,

Shan,

Yang

et al. 2022

Preprint

View full text Add to dashboard Cite

In large-scale online services, crucial metrics, a.k.a., key performance indicators (KPIs), are monitored periodically to check their running statuses. Generally, KPIs are aggregated along multiple dimensions and derived by complex calculations among fundamental metrics from the raw data. Once abnormal KPI values are observed, root cause analysis (RCA) can be applied to identify the reasons for anomalies, so that we can troubleshoot quickly. Recently, several automatic RCA techniques were proposed to localize the related dimensions (or a combination of dimensions) to explain the anomalies. However, their analyses are limited to the data on the abnormal metric and ignore the data of other metrics which may be also related to the anomalies, leading to imprecise or even incorrect root causes. To this end, we propose a cross-metric multi-dimensional root cause analysis method, named CMMD, which consists of two key components: 1) relationship modeling, which utilizes graph neural network (GNN) to model the unknown complex calculation among metrics and aggregation function among dimensions from historical data; 2) root cause localization, which adopts the genetic algorithm to efficiently and effectively dive into the raw data and localize the abnormal dimension(s) once the KPI anomalies are detected. Experiments on synthetic datasets, public datasets and online production environment demonstrate the superiority of our proposed CMMD method compared with baselines. Currently, CMMD is running as an online service in Microsoft Azure.

show abstract

Section: Root Cause Analysismentioning

confidence: 99%

CMMD: Cross-Metric Multi-Dimensional Root Cause Analysis

Yan,

Shan,

Yang

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Finally, a different line of research studies how to predict incidents using texts, issue reports, and statistical data [9], [11]. Differently, in this paper we studied the challenge of anticipating failures based on the KPIs collected online.…”

Section: B Failure Prediction In Cloud Environmentsmentioning

confidence: 99%

Cloud Failure Prediction with Hierarchical Temporal Memory: An Empirical Assessment

Riganelli¹,

Saltarel²,

Tundo³

et al. 2021

Preprint

View full text Add to dashboard Cite

Hierarchical Temporary Memory (HTM) is an unsupervised learning algorithm inspired by the features of the neocortex that can be used to continuously process stream data and detect anomalies, without requiring a large amount of data for training nor requiring labeled data. HTM is also able to continuously learn from samples, providing a model that is always up-to-date with respect to observations. These characteristics make HTM particularly suitable for supporting online failure prediction in cloud systems, which are systems with a dynamically changing behavior that must be monitored to anticipate problems. This paper presents the first systematic study that assesses HTM in the context of failure prediction.The results that we obtained considering 72 configurations of HTM applied to 12 different types of faults introduced in the Clearwater cloud system show that HTM can help to predict failures with sufficient effectiveness (F-measure = 0.76), representing an interesting practical alternative to (semi-)supervised algorithms.

show abstract

“…Thus, incident management becomes a hotspot topic in both academia and industry. Massive amount of effort has been devoted to incident detection [15], [37]- [39] and incident triage [38], [40]- [42]. For example, Lim et al [43] utilized Hidden Markov Random Field for performance issue clustering to identify representative issues.…”

Section: B Incident Managementmentioning

confidence: 99%

Continuous Incident Triage for Large-Scale Online Service Systems

Chen

Lin

et al. 2019

2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE)

Self Cite

View full text Add to dashboard Cite

As online service systems continue to grow in terms of complexity and volume, how service incidents are managed will significantly impact company revenue and user trust. Due to the cascading effect, cloud failures often come with an overwhelming number of incidents from dependent services and devices. To pursue efficient incident management, related incidents should be quickly aggregated to narrow down the problem scope. To this end, in this paper, we propose GRLIA, an incident aggregation framework based on graph representation learning over the cascading graph of cloud failures. A representation vector is learned for each unique type of incident in an unsupervised and unified manner, which is able to simultaneously encode the topological and temporal correlations among incidents. Thus, it can be easily employed for online incident aggregation. In particular, to learn the correlations more accurately, we try to recover the complete scope of failures' cascading impact by leveraging fine-grained system monitoring data, i.e., Key Performance Indicators (KPIs). The proposed framework is evaluated with real-world incident data collected from a large-scale online service system of Huawei Cloud. The experimental results demonstrate that GRLIA is effective and outperforms existing methods. Furthermore, our framework has been successfully deployed in industrial practice.

show abstract

Efficient incident identification from multi-dimensional issue reports via meta-heuristic search

Cited by 22 publications

References 38 publications

CMMD: Cross-Metric Multi-Dimensional Root Cause Analysis

CMMD: Cross-Metric Multi-Dimensional Root Cause Analysis

Cloud Failure Prediction with Hierarchical Temporal Memory: An Empirical Assessment

Continuous Incident Triage for Large-Scale Online Service Systems

Contact Info

Product

Resources

About