Continuous Incident Triage for Large-Scale Online Service Systems

Abstract

As online service systems continue to grow in terms of complexity and volume, how service incidents are managed will significantly impact company revenue and user trust. Due to the cascading effect, cloud failures often come with an overwhelming number of incidents from dependent services and devices. To pursue efficient incident management, related incidents should be quickly aggregated to narrow down the problem scope. To this end, in this paper, we propose GRLIA, an incident aggregation framework based on graph representation learning over the cascading graph of cloud failures. A representation vector is learned for each unique type of incident in an unsupervised and unified manner, which is able to simultaneously encode the topological and temporal correlations among incidents. Thus, it can be easily employed for online incident aggregation. In particular, to learn the correlations more accurately, we try to recover the complete scope of failures' cascading impact by leveraging fine-grained system monitoring data, i.e., Key Performance Indicators (KPIs). The proposed framework is evaluated with real-world incident data collected from a large-scale online service system of Huawei Cloud. The experimental results demonstrate that GRLIA is effective and outperforms existing methods. Furthermore, our framework has been successfully deployed in industrial practice.

Efficient customer incident triage via linking with system incidents

“…It is worth noting that there is a body of work focuses on incident triage [8,9]. Chen et al [8] conducts a comprehensive empirical study of incident triage on 20 real-world online service systems.…”

“…This work also tries the practicability of bug triage approaches on incident triage problems, and it concludes that traditional bug triage methods need to be further improved to fit the context of incident triage. Chen et al [9] further propose DeepCT, a deep learning based approach which extracts features from historical discussions and thus solves the continuous incident triage problem. However, these work could only help back-end incident mitigation in largescale systems.…”

“…The insufficient data makes it hard to directly apply machine learning techniques. Existing approaches either focus on issue detection using customer feedback [50] or directly conduct incident triage based on system incident reports [9]. Currently, there is no approach that attempts to link CI with MI.…”

Deep learning library testing via effective model generation

“…In recent years, deep learning (DL) techniques are rapidly developed and become one of the most popular techniques. Also, they are widely adopted in various domains in practice, such as autonomous driving cars [12], face recognition [59], speech recognition [29], aircraft collision avoidance systems [36], and software engineering [15,16,18,21,42,70]. Unfortunately, DL systems are also shown to be vulnerable to attacks and lack of robustness [40,67].…”

Real-time incident prediction for online service systems

“…To reduce the influence of incidents and guarantee the quality of software services, there are two widely-used ways in both academia and industry [32,33], i.e., predicting the occurrence of an incident in advance so that engineers can take some proactive actions to prevent it [18,43] and mitigate the already happened incident as soon as possible [14,15]. Our work focuses on the first way since this way is able to directly avoid the occurrence of service unavailability rather than reduce the time of service unavailability.…”