Operating System Level Trace Analysis for Automated Problem Identification

Abstract

Performance bottlenecks, malicious activities, programming bugs and other kinds of problematic behavior could be accurately detected on production systems if the relevant events were being monitored. This could be achieved through kernel level tracing where every time a relevant event occurs, the information is analysed or saved in a trace file to be inspected during post-mortem analysis. While collecting the information from the kernel has a very low impact, the offline analysis is typically performed remotely with no overhead on the system whatsoever.This article presents an automata-based approach for analyzing traces generated by the kernel of an operating system. Some typical patterns of problematic behavior are identified and described using the State Machine Language. These patterns are fed into an offline analyzer which efficiently and simultaneously checks for their occurrences even in traces of several gigabytes. The analyzer achieves a linear performance with respect to the trace size. The remaining factors impacting its performance are also discussed. The main interest of the proposed approach is the efficiency obtained in monitoring such extensive and detailed execution traces for a very large number of simultaneous possible patterns of problematic behavior.

A Flexible Data-Driven Approach for Execution Trace Filtering

“…However, we can discuss the criteria that affect the performance and suggest potential optimizations. In a similar previous work [6], mentioned in the literature, Gabriel Matni et al [6] discussed the performance of their solutions. The criteria they listed are similar to ours.…”

“…Various papers have expressed opinions and findings on pattern description languages in the past filteringmalony, filteringsudanshu, fadel, wahab-correlation, ezzati. In [6], [8], the authors divided these description languages into several categories and give examples for each category. Their solutions, in general, are not a stateful approach, which would preserve the common and recent status of the analyzer and would avoid recalculating and regenerating this status frequently [3], [9].…”

“…They provide details on the syntax of the language before giving use-cases of attacks. Gabriel Matni et al explain in [6] the interest of choosing a state machine approach to describe patterns. They express the benefits of this choice based on the "simplicity and expressiveness" of the statetransition language ( [6]), the domain independence advantage and the ease to generate synthetic events.…”

“…Gabriel Matni et al explain in [6] the interest of choosing a state machine approach to describe patterns. They express the benefits of this choice based on the "simplicity and expressiveness" of the statetransition language ( [6]), the domain independence advantage and the ease to generate synthetic events. Wasseem Fadel in [7] and also Naser Ezzati et al in [3] generate synthetic events to realize trace abstraction, using state machines to describe patterns.…”

“…There are many techniques in the literature that use pattern based techniques to abstract or filter out the trace data [6], [7], [8]. However, they usually use a constant set of patterns, mostly hard-coded in the system, which reduces the flexibility of the approach.…”