Performance analysis of distributed storage clusters based on kernel and userspace traces

Abstract

Distributed storage systems are commonly used in modern computing. They are highly scalable and offer data replication and fault tolerance. The complexity of those systems makes them difficult to debug using traditional tools. The existing tools are able to evaluate the overall performance of such systems but they do not provide enough information to find the root cause of performance issues. In this article, we propose a tracing-based performance analysis framework for storage clusters. We use a tracing strategy that reduces the tracing overhead in production systems. The traces collected from the different storage nodes are correlated and used to generate a data model that represents the cluster. Userspace tracing is used to gather data from the storage daemons, while Kernel tracing is used to provide detailed information about operating system internals such as disk queues, network queues and process scheduling. Efficient data structures are used to store the model and to generate metrics and graphical views. Our tool is used in different real world scenarios and is able to investigate interesting performance problems including I/O latencies, data replication and storage nodes failures.

Toward a Practical and Timely Diagnosis of Application’s I/O Behavior

“…Therefore, these do not provide the flexibility to implement custom analysis algorithms nor enable users to and explore other information contained in the collected I/O traces. On the other hand, solutions similar to DIO that support customizable analysis fail to capture relevant information to diagnose the use cases discussed in this paper [3], [39].…”

“…Specifically, the server log file is repeatedly opened and closed for every written line, which adds extra latency for log operations and can potentially slow down Redis performance. 3 To identify this behavior, users could run a workload on top of Redis and trace the syscalls submitted to kernel. In this example, we used redis-benchmark [24] to generate 5M requests to the database, which yield >200M syscalls.…”

“…[15]- [17]. Other tools provide modules for automating the analysis of traced data but follow an offline approach, where this data needs to be stored first and, only later, it is parsed and provided as input to the analysis pipeline [3], [4], [18], [39]. Only DIO and Longline [19] automatically parse and forward traced events to the analysis pipeline by following an inline (near real-time) approach.…”

“…Moreover, DIO offers predefined representations that automatically summarize and allow the visualization of the I/O patterns discussed in the paper. Moreover, our tool enables users to create new visualizations commonly supported by other diagnosis solutions (e.g., tables, pie charts, histograms, heatmaps, time series) [3], [19], [39].…”

“…The sheer amount of storage operations generated by these applications, ranging from hundreds to thousands of operations per second, makes their analysis a complex and timeconsuming task when done manually. Thus, diagnosis tools that can help users and developers profile more precisely the I/O interaction between applications and corresponding storage backends are crucial for debugging errors, finding performance and dependability issues, and identifying potential optimizations for applications [3]- [5].…”