An Approach for Anomaly Diagnosis Based on Hybrid Graph Model with Logs for Distributed Services

Jia, Tong; Chen, Pengfei; Yang, Lin; Liu, Ying; Meng, Fan Jing; Xu, Jie

doi:10.1109/icws.2017.12

Cited by 58 publications

(16 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Xu et al [23] introduced LogDC, a log model based problem diagnosis tool for cloud applications with the full-lifecycle Kubernetes logs. Jia et al [24] proposed an approach for automatic anomaly detection based on logs. It raises anomaly alerts on observing deviations from the hybrid model which captures normal execution flows inter and intra services.…”

Section: E Threats To Validitymentioning

confidence: 99%

MicroHECL: High-Efficient Root Cause Localization in Large-Scale Microservice Systems

Liu

Peng

et al. 2021

2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)

View full text Add to dashboard Cite

Availability issues of industrial microservice systems (e.g., drop of successfully placed orders and processed transactions) directly affect the running of the business. These issues are usually caused by various types of service anomalies which propagate along service dependencies. Accurate and high-efficient root cause localization is thus a critical challenge for large-scale industrial microservice systems. Existing approaches use service dependency graph based analysis techniques to automatically locate root causes. However, these approaches are limited due to their inaccurate detection of service anomalies and inefficient traversing of service dependency graph. In this paper, we propose a high-efficient root cause localization approach for availability issues of microservice systems, called MicroHECL. Based on a dynamically constructed service call graph, MicroHECL analyzes possible anomaly propagation chains, and ranks candidate root causes based on correlation analysis. We combine machine learning and statistical methods and design customized models for the detection of different types of service anomalies (i.e., performance, reliability, traffic). To improve the efficiency, we adopt a pruning strategy to eliminate irrelevant service calls in anomaly propagation chain analysis. Experimental studies show that MicroHECL significantly outperforms two state-of-the-art baseline approaches in terms of both accuracy and efficiency. MicroHECL has been used in Alibaba and achieves a top-3 hit ratio of 68% with root cause localization time reduced from 30 minutes to 5 minutes.

show abstract

Section: E Threats To Validitymentioning

confidence: 99%

MicroHECL: High-Efficient Root Cause Localization in Large-Scale Microservice Systems

Liu

Peng

et al. 2021

2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)

View full text Add to dashboard Cite

show abstract

“…The existing anomaly detection methods based on log data are mainly classified into three categories, which are graph model-based [36][37][38], probability analysis-based [39] and machine learning based detection methods [40]. Anomaly detection based on graph is used to model the sequence relationship, association relationship, and log text content.…”

Section: Anomaly Detectionmentioning

confidence: 99%

ConAnomaly: Content-Based Anomaly Detection for System Logs

Luktarhan

Chen

2021

Sensors

View full text Add to dashboard Cite

Enterprise systems typically produce a large number of logs to record runtime states and important events. Log anomaly detection is efficient for business management and system maintenance. Most existing log-based anomaly detection methods use log parser to get log event indexes or event templates and then utilize machine learning methods to detect anomalies. However, these methods cannot handle unknown log types and do not take advantage of the log semantic information. In this article, we propose ConAnomaly, a log-based anomaly detection model composed of a log sequence encoder (log2vec) and multi-layer Long Short Term Memory Network (LSTM). We designed log2vec based on the Word2vec model, which first vectorized the words in the log content, then deleted the invalid words through part of speech tagging, and finally obtained the sequence vector by the weighted average method. In this way, ConAnomaly not only captures semantic information in the log but also leverages log sequential relationships. We evaluate our proposed approach on two log datasets. Our experimental results show that ConAnomaly has good stability and can deal with unseen log types to a certain extent, and it provides better performance than most log-based anomaly detection methods.

show abstract

“…The system log is an important clue for analysis [2]. By parsing patterns and extracting features from event logs, Xu et al [15,16] built anomaly detection and identification models from historical data and used these models to analyze root causes. However, as the application flexibility increases, these methods are less effective in analyzing the anomalies in real-time.…”

Section: Related Workmentioning

confidence: 99%

AAMR: Automated Anomalous Microservice Ranking in Cloud-Native Environment

Zhang¹

2021

Proceedings of the 33rd International Conference on Software Engineering and Knowledge Engineering

View full text Add to dashboard Cite

Recently, it has become a trend for developers to build applications using the microservice architecture. The functionality of each application is divided into multiple independent microservices, which are interconnected to others. With the emergence of cloud-native technologies, such as Docker and Kubernetes, developers can achieve a consistent and scalable delivery for complex software applications. However, it is challenging to diagnose performance issues in microservices due to the complex runtime environments and the numerous metrics. In this paper, we propose a novel root cause analysis approach named AAMR. AAMR firstly constructs a service dependency graph based on real-time metrics. Next, it updates the anomaly weight of each microservice automatically. Finally, a PageRankbased random walk is applied for further ranking root causes, i.e., ranking potential problematic services. Experiments conducted on Kubernetes clusters show that the proposed approach achieves a good analysis result, which outperforms several state-of-the-art methods.

show abstract

An Approach for Anomaly Diagnosis Based on Hybrid Graph Model with Logs for Distributed Services

Cited by 58 publications

References 15 publications

MicroHECL: High-Efficient Root Cause Localization in Large-Scale Microservice Systems

MicroHECL: High-Efficient Root Cause Localization in Large-Scale Microservice Systems

ConAnomaly: Content-Based Anomaly Detection for System Logs

AAMR: Automated Anomalous Microservice Ranking in Cloud-Native Environment

Contact Info

Product

Resources

About