Combining Distributed and Kernel Tracing for Performance Analysis of Cloud Applications

Abstract

Distributed tracing allows tracking user requests that span across multiple services and machines in a distributed application. However, typical cloud applications rely on abstraction layers that can hide the root cause of latency happening between processes or in the kernel. Because of its focus on high-level events, existing methodologies in applying distributed tracing can be limited when trying to detect complex contentions and relate them back to the originating requests. Cross-level analyses that include kernel-level events are necessary to debug problems as prevalent as mutex or disk contention, however cross-level analysis and associating events in the kernel and distributed tracing data is complex and can add a lot of overhead. This paper describes a new solution for combining distributed tracing with low-level software tracing in order to find the latency root cause better. We explain how we achieve a hybrid trace collection to capture and synchronize both kernel and distributed request events. Then, we present our design and implementation for a critical path analysis. We show that our analysis describes precisely how each request spends its time and what stands in its critical path while limiting overhead.

DTraComp: Comparing distributed execution traces for understanding intermittent latency sources

“…The techniques mentioned above are specifically suitable for distributed systems because they emphasize context propagation and large-scale tracing. However, it is essential to note that these techniques are limited to capturing high-level events [7] and cannot extract system information related to performance degradation. Researchers [19,20], who work on low-level traces to detect or diagnose software performance degradation, encounter a large volume of events that they need to collect with high accuracy and low overhead.…”

“…Since low-level and high-level software tracing offer unique advantages, hybrid approaches have been proposed to make the best of both worlds. Gelle et al [7] suggest a solution that combines kernel tracing with distributed tracing, to better scrutinize events and determine the underlying cause of performance issues. By merging the advantages of both approaches, this method can obtain precise information about system interactions and high-level events, from distributed tracing, while collecting detailed and specific low-level events from kernel and user space tracing.…”

“…It excels at high-throughput event collection, with a low impact on the target application. However, software tracing is not designed for end-to-end tracing in distributed environments [7].…”

Provenance-enhanced Root Cause Analysis for Jupyter Notebooks

“…For distributed tracing, one of the most basic forms is to create a methodology to compare the traces visually [34]. Gelle et al [37] included kernel level events in the high-level logs to the tracing to improve the detection of anomalies and relate them to their root cause. Bento et al [38] went from visualizing tracing data for human inspection to automating the analysis of tracing data.…”

Vnode: Low-Overhead Transparent Tracing of Node.js-Based Microservice Architectures

“…Additionally, arranging the components in a way that maintains system attributes such as availability and low latency introduces intricacies [3,4]. The challenges of debugging [5][6][7][8][9] and component arrangement [10] have been addressed through the implementation of distributed tracing techniques [11][12][13][14][15][16][17][18]. However, applying these methods necessitates instrumenting the application's source code, which brings additional overhead and the risk of altering the application's behavior.…”