Automatically and Adaptively Identifying Severe Alerts for Online Service Systems

Zhao, Ning; Jin, Panshi; Wang, Lixin; Yang, Xiaoqin; Liu, Rong; Zhang, Wenchi; Sui, Kaixin; Pei, Dan

doi:10.1109/infocom41043.2020.9155219

Cited by 22 publications

(13 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Zhao et al [6] reported an approach for handling alert storms consisting of alert storm detection using Extreme Value Theory (EVT), alert filtering using ML Isolation Forest method, alert clustering using Similarity Matrix Construction, and representative alert selection. Furthermore, Zhao et al [17] published another study on enhancing the quality of services by utilizing the monitoring data. Similarly, they analyzed alerts but with aim of identifying the severity level.…”

Section: Background and Related Workmentioning

confidence: 99%

“…Hence, we improve the feedback loop from operations to development by introducing a new element, a smart filter, for optimization of alert to noise ratio. In the design process, we considered the insights gained through interviews, results of the intensive discussions with the development team, and state of the art solutions for alert management [6,17].…”

Section: Research Approachmentioning

confidence: 99%

See 1 more Smart Citation

Closing the Feedback Loop in DevOps Through Autonomous Monitors in Operations

2021

View full text Add to dashboard Cite

DevOps represent the tight connection between development and operations. To address challenges that arise on the borderline between development and operations, we conducted a study in collaboration with a Swedish company responsible for ticket management and sales in public transportation. The aim of our study was to explore and describe the existing DevOps environment, as well as to identify how the feedback from operations can be improved, specifically with respect to the alerts sent from system operations. Our study complies with the basic principles of the design science paradigm, such as understanding and improving design solutions in the specific areas of practice. Our diagnosis, based on qualitative data collected through interviews and observations, shows that alert flooding is a challenge in the feedback loop, i.e. too much signals from operations create noise in the feedback loop. Therefore, we design a solution to improve the alert management by optimizing when to raise alerts and accordingly introducing a new element in the feedback loop, a smart filter. Moreover, we implemented a prototype of the proposed solution design and showed that a tighter relation between operations and development can be achieved, using a hybrid method which combines rule-based and unsupervised machine learning for operations data analysis.

show abstract

Section: Background and Related Workmentioning

confidence: 99%

Section: Research Approachmentioning

confidence: 99%

Closing the Feedback Loop in DevOps Through Autonomous Monitors in Operations

2021

View full text Add to dashboard Cite

show abstract

“…Many existing work [4], [44] address this problem by reducing the duplicated or correlated alerts. For example, Zhao et al [45] aimed to recommend the severe alerts to engineers. Lin et al [39] proposed an alert correlation method to cluster semi-structured alert texts to gain insights from the clustering results.…”

Section: B Incident Managementmentioning

confidence: 99%

Continuous Incident Triage for Large-Scale Online Service Systems

Chen

Lin

et al. 2019

2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE)

View full text Add to dashboard Cite

As online service systems continue to grow in terms of complexity and volume, how service incidents are managed will significantly impact company revenue and user trust. Due to the cascading effect, cloud failures often come with an overwhelming number of incidents from dependent services and devices. To pursue efficient incident management, related incidents should be quickly aggregated to narrow down the problem scope. To this end, in this paper, we propose GRLIA, an incident aggregation framework based on graph representation learning over the cascading graph of cloud failures. A representation vector is learned for each unique type of incident in an unsupervised and unified manner, which is able to simultaneously encode the topological and temporal correlations among incidents. Thus, it can be easily employed for online incident aggregation. In particular, to learn the correlations more accurately, we try to recover the complete scope of failures' cascading impact by leveraging fine-grained system monitoring data, i.e., Key Performance Indicators (KPIs). The proposed framework is evaluated with real-world incident data collected from a large-scale online service system of Huawei Cloud. The experimental results demonstrate that GRLIA is effective and outperforms existing methods. Furthermore, our framework has been successfully deployed in industrial practice.

show abstract

“…In the process of recovering a system, it is critical to conduct accurate and efficient root cause analysis (RCA) [2], the second one of a three-step process. In the first step, anomalies are detected with alerting mechanisms [3]- [5] based on monitoring data such as logs [6]- [10], metrics/key performance indicators (KPIs) [11]- [15], or a combination thereof [16], [17]. In the second step, when the alerts are triggered, RCA is performed to analyze the root cause of these and additional events and propose recovery actions from the associated incident [6], [18], [19].…”

Section: Introductionmentioning

confidence: 99%

“…To overcome the limited effectiveness of existing approaches [2], [3], [14], [16], [21]- [31] (as mentioned in Section II) in industrial settings due to the aforementioned complexities, we propose GROOT, an event-graph-based RCA approach. In particular, GROOT constructs an event causality graph, the basic nodes are monitoring events such as performance metrics deviation events, status change event and developer activity events.…”

Section: Introductionmentioning

confidence: 99%

Groot: An Event-graph-based Approach for Root Cause Analysis in Industrial Settings

Wang¹,

Wu²,

Jiang³

et al. 2021

Preprint

View full text Add to dashboard Cite

For large-scale distributed systems, it is crucial to efficiently diagnose the root causes of incidents to maintain high system availability. The recent development of microservice architecture brings three major challenges (i.e., operation, system scale, and monitoring complexities) to root cause analysis (RCA) in industrial settings. To tackle these challenges, in this paper, we present GROOT, an event-graph-based approach for RCA. GROOT constructs a real-time causality graph based on events that summarize various types of metrics, logs, and activities in the system under analysis. Moreover, to incorporate domain knowledge from site reliability engineering (SRE) engineers, GROOT can be customized with user-defined events and domainspecific rules. Currently, GROOT supports RCA among 5,000 real production services and is actively used by the SRE team in a global e-commerce system serving more than 185 million active buyers per year. Over 15 months, we collect a data set containing labeled root causes of 952 real production incidents for evaluation. The evaluation results show that GROOT is able to achieve 95% top-3 accuracy and 78% top-1 accuracy. To share our experience in deploying and adopting RCA in industrial settings, we conduct survey to show that users of GROOT find it helpful and easy to use. We also share the lessons learned from deploying and adopting GROOT to solve RCA problems in production environments.

show abstract

Automatically and Adaptively Identifying Severe Alerts for Online Service Systems

Cited by 22 publications

References 33 publications

Closing the Feedback Loop in DevOps Through Autonomous Monitors in Operations

Closing the Feedback Loop in DevOps Through Autonomous Monitors in Operations

Continuous Incident Triage for Large-Scale Online Service Systems

Groot: An Event-graph-based Approach for Root Cause Analysis in Industrial Settings

Contact Info

Product

Resources

About