TraceRank: Abnormal service localization with dis‐aggregated end‐to‐end tracing data in cloud native systems

Abstract

Modern cloud native applications are generally built with a microservice architecture. To tackle various performance problems among a large number of services and machines, an end‐to‐end tracing tool is always equipped in these systems to track the execution path of every single request. However, it is nontrivial to conduct root cause analysis of anomalies with such a large volume of tracing data. This paper proposes a novel system named TraceRank to identify and locate abnormal services causing performance problems with dis‐aggregated end‐to‐end traces. TraceRank mainly includes an anomaly detection module and a root cause analysis module. The root cause analysis procedure is triggered when an anomaly is detected. To fully leverage the information provided by the tracing data, both the spectrum analysis and the PageRank‐based random walk methods are introduced to pinpoint abnormal services. The experiments in TrainTicket and Bookinfo microservice benchmarks and a real‐world system show that TraceRank can locate root causes with 90% in Precision and 86% in Recall. TraceRank has up to 10% improvement compared with several state‐of‐the‐art approaches in both Precision and Recall. Finally, TraceRank has good scalability and a low overhead to adapt to large‐scale microservice systems.

MetricSifter: Feature Reduction of Multivariate Time Series Data for Efficient Fault Localization in Cloud Applications

“…utilized in many related works [10], [11], [15], [21], [25], [63]. Sock Shop consists of seven microservices, and Train Ticket consists of 41 microservices, positioning it as one of the largest public microservices benchmarks.…”

Graph-Based Root Cause Localization in Microservice Systems with Protection Mechanisms

“…Trace Anomaly Score: The Spectrum-based fault localization (SBFL) technique 25,26 has been widely used in software testing, which is mainly based on the coverage information of current program elements to assess the degree of program anomaly. Inspired by this, we calculate the anomaly score by using the SBFL technique, which is also mentioned in previous approaches 9,11,27,28 .…”

“…In this section, we use the PageRank algorithm to locate root cause of abnormal services, which has proven good performance in anomaly propagation and root cause localization 11,13,32,33 . The PageRank algorithm is a well-known approach for web analysis that aims to rank the importance of web pages.…”

“…MEPFL locates the root causes by training a supervised learning model, but it is impractical to guarantee accuracy in different microservice environments. On the other hand, the service dependency graph approaches use causal relationships [10][11][12] or service calls [13][14][15] to construct service dependency graphs. These approaches use metrics, logs and traces to calculate service anomaly scores and locate the root cause of abnormal services by traversing the service dependency graph.…”

Model-based Resource Management for Fine-grained Services

“…This is significant problem for a data center that hosts microservices in the scale of thousands. Data centers like this can produce tens of terabytes of trace data per day [YHC21], requiring a huge storage capacity. A major challenge in distributed tracing is to reduce this storage requirement by sampling the traces that can be potentially useful for further analysis [LCPAM19].…”