Production-run software failure diagnosis via hardware performance counters

Abstract

Sequential and concurrency bugs are widespread in deployed software. They cause severe failures and huge financial loss during production runs. Tools that diagnose production-run failures with low overhead are needed. The state-of-the-art diagnosis techniques use software instrumentation to sample program properties at run time and use off-line statistical analysis to identify properties most correlated with failures. Although promising, these techniques suffer from high run-time overhead, which is sometimes over 100%, for concurrency-bug failure diagnosis and hence are not suitable for production-run usage.We present PBI, a system that uses existing hardware performance counters to diagnose production-run failures caused by sequential and concurrency bugs with low overhead. PBI is designed based on several key observations. First, a few widely supported performance counter events can reflect a wide variety of common software bugs and can be monitored by hardware with almost no overhead. Second, the counter overflow interrupt supported by existing hardware and operating systems provides a natural and effective mechanism to conduct event sampling at user level. Third, the noise and non-determinism in interrupt delivery complements well with statistical processing.We evaluate PBI using 13 real-world concurrency and sequential bugs from representative open-source server, client, and utility programs, and 10 bugs from a widely used software-testing benchmark. Quantitatively, PBI can effectively diagnose failures caused by these bugs with a small overhead that is never higher than 10 %. Qualitatively, PBI does not require any change to software and presents a novel use of existing hardware performance counters.

Statistical debugging for real-world performance problems

“…The basic statistical model that works for most functional bugs [4,6,17,19,22,23,33] is very useful for performance diagnosis too, but still leaves many performance problems uncovered; statistical models that consider the number of times a predicate is true in each run (e.g., the ∆LDA model) is needed for diagnosing performance problems.…”

“…Different predicates are designed for statistical debugging that targets different types of functional bugs. For example, branch predicates, function-return predicates, and others are used for diagnosing sequential bugs [22,23]; interleaving-related predicates are used for diagnosing concurrency bugs [6,17].…”

“…In general, statistical debugging [4,6,17,19,22,23,33] is an approach that uses statistical machine learning techniques to help failure diagnosis. It usually works in two steps.…”

“…Most previous work on functional bug diagnosis has found the basic model to be sufficient [6,17,22,23] and did not try ∆LDA style models.…”

“…Previous work on production-run functional bug diagnosis [6,17,22,23] proposes to achieve these two goals by combining sampling techniques with statistical debugging. By randomly sampling predicates at run time, the overhead can be significantly decreased; by processing predicates collected from many failure and success runs together, the diagnosis capability can be maintained for the diagnosis of most functional bugs [6,17,22,23].…”