MicroHECL: High-Efficient Root Cause Localization in Large-Scale Microservice Systems

Liu, Dewei; He, Chuan; Peng, Xin; Lin, Fan; Zhang, Chenxi; Gong, Shengfang; Li, Ziang; Ou, Jiayu; Wu, Zheshun

doi:10.1109/icse-seip52600.2021.00043

Cited by 67 publications

(26 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently there have been works on rootcause analysis or anomaly localization in Micro-service based systems, e.g. [8], [9] where the goal is to find the failing microservice among the services.…”

Section: Introductionmentioning

confidence: 99%

Pinpointing Anomaly Events in Logs from Stability Testing -- N-Grams vs. Deep-Learning

Mäntylä¹,

Varela²,

Hashemi³

2022

Preprint

View full text Add to dashboard Cite

As stability testing execution logs can be very long, software engineers need help in locating anomalous events. We develop and evaluate two models for scoring individual logevents for anomalousness, namely an N-Gram model and a Deep Learning model with LSTM (Long short-term memory). Both are trained on normal log sequences only. We evaluate the models with long log sequences of Android stability testing in our company case and with short log sequences from HDFS (Hadoop Distributed File System) public dataset. We evaluate next event prediction accuracy and computational efficiency. The LSTM model is more accurate in stability testing logs (0.848 vs 0.865), whereas in HDFS logs the N-Gram is slightly more accurate (0.904 vs 0.900). The N-Gram model has far superior computational efficiency compared to the Deep model (4 to 13 seconds vs 16 minutes to nearly 4 hours), making it the preferred choice for our case company. Scoring individual log events for anomalousness seems like a good aid for root cause analysis of failing test cases, and our case company plans to add it to its online services. Despite the recent surge in using deep learning in software system anomaly detection, we found limited benefits in doing so. However, future work should consider whether our finding holds with different LSTM-model hyper-parameters, other datasets, and with other deep-learning approaches that promise better accuracy and computational efficiency than LSTM based models.

show abstract

Section: Introductionmentioning

confidence: 99%

Pinpointing Anomaly Events in Logs from Stability Testing -- N-Grams vs. Deep-Learning

Mäntylä¹,

Varela²,

Hashemi³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Multivariate time series serves as an important reference of the "health status" of many Internet applications such cloud computing, micro-service systems and the Internet routing network, to name a few. For example, consider the cloud computing system, which has become a crucial infrastructure for many customized services of companies and governments [22,26,19]. To maintain a high quality of service of the cloud computing system, it is important to monitor the "health status" of the cloud system and perform troubleshooting in real-time [19,22,26].…”

Section: Introductionmentioning

confidence: 99%

“…For example, consider the cloud computing system, which has become a crucial infrastructure for many customized services of companies and governments [22,26,19]. To maintain a high quality of service of the cloud computing system, it is important to monitor the "health status" of the cloud system and perform troubleshooting in real-time [19,22,26]. To achieve this, many performance metrics of a cloud computing system such as CPU usage, disk I/O rate, network packet loss rate, etc., are monitored in real-time, which form a MTS [19,22,26].…”

Section: Introductionmentioning

confidence: 99%

“…To maintain a high quality of service of the cloud computing system, it is important to monitor the "health status" of the cloud system and perform troubleshooting in real-time [19,22,26]. To achieve this, many performance metrics of a cloud computing system such as CPU usage, disk I/O rate, network packet loss rate, etc., are monitored in real-time, which form a MTS [19,22,26]. From the MTS, system anomalies can be detected via machine learning [4,12,17,23,37,42].…”

Section: Introductionmentioning

confidence: 99%

“…This makes it difficult to learn the normal and abnormal pattern from the MTS. Fourth, the MTS data is usually nonstationary, [19] making it difficult to learn stable patterns. In this paper, we propose an algorithm called LPC-AD to address these challenges.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

LPC-AD: Fast and Accurate Multivariate Time Series Anomaly Detection via Latent Predictive Coding

Qi¹,

Xie²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper proposes LPC-AD, a fast and accurate multivariate time series (MTS) anomaly detection method. LPC-AD is motivated by the ever-increasing needs for fast and accurate MTS anomaly detection methods to support fast troubleshooting in cloud computing, micro-service systems, etc. LPC-AD is fast in the sense that its reduces the training time by as high as 38.2% compared to the state-of-the-art (SOTA) deep learning methods that focus on training speed. LPC-AD is accurate in the sense that it improves the detection accuracy by as high as 18.9% compared to SOTA sophisticated deep learning methods that focus on enhancing detection accuracy. Methodologically, LPC-AD contributes a generic architecture LPC-Reconstruct for one to attain different trade-offs between training speed and detection accuracy. More specifically, LPC-Reconstruct is built on ideas from autoencoder for reducing redundancy in time series, latent predictive coding for capturing temporal dependence in MTS, and randomized perturbation for avoiding overfitting of anomalous dependence in the training data. We present simple instantiations of LPC-Reconstruct to attain fast training speed, where we propose a simple randomized perturbation method. The superior performance of LPC-AD over SOTA methods is validated by extensive experiments on four large real-world datasets. Experiment results also show the necessity and benefit of each component of the LPC-Reconstruct architecture and that LPC-AD is robust to hyper parameters.

show abstract

TraceRank: Abnormal service localization with dis‐aggregated end‐to‐end tracing data in cloud native systems

Huang

Chen

2021

J Software Evolu Process

View full text Add to dashboard Cite

Modern cloud native applications are generally built with a microservice architecture. To tackle various performance problems among a large number of services and machines, an end‐to‐end tracing tool is always equipped in these systems to track the execution path of every single request. However, it is nontrivial to conduct root cause analysis of anomalies with such a large volume of tracing data. This paper proposes a novel system named TraceRank to identify and locate abnormal services causing performance problems with dis‐aggregated end‐to‐end traces. TraceRank mainly includes an anomaly detection module and a root cause analysis module. The root cause analysis procedure is triggered when an anomaly is detected. To fully leverage the information provided by the tracing data, both the spectrum analysis and the PageRank‐based random walk methods are introduced to pinpoint abnormal services. The experiments in TrainTicket and Bookinfo microservice benchmarks and a real‐world system show that TraceRank can locate root causes with 90% in Precision and 86% in Recall. TraceRank has up to 10% improvement compared with several state‐of‐the‐art approaches in both Precision and Recall. Finally, TraceRank has good scalability and a low overhead to adapt to large‐scale microservice systems.

show abstract

MicroHECL: High-Efficient Root Cause Localization in Large-Scale Microservice Systems

Cited by 67 publications

References 18 publications

Pinpointing Anomaly Events in Logs from Stability Testing -- N-Grams vs. Deep-Learning

Pinpointing Anomaly Events in Logs from Stability Testing -- N-Grams vs. Deep-Learning

LPC-AD: Fast and Accurate Multivariate Time Series Anomaly Detection via Latent Predictive Coding

TraceRank: Abnormal service localization with dis‐aggregated end‐to‐end tracing data in cloud native systems

Contact Info

Product

Resources

About