Production-run software failure diagnosis via hardware performance counters

Abstract

Sequential and concurrency bugs are widespread in deployed software. They cause severe failures and huge financial loss during production runs. Tools that diagnose production-run failures with low overhead are needed. The state-of-the-art diagnosis techniques use software instrumentation to sample program properties at run time and use off-line statistical analysis to identify properties most correlated with failures. Although promising, these techniques suffer from high run-time overhead, which is sometimes over 100%, for concurrency-bug failure diagnosis and hence are not suitable for production-run usage. We present PBI, a system that uses existing hardware performance counters to diagnose production-run failures caused by sequential and concurrency bugs with low overhead. PBI is designed based on several key observations. First, a few widely supported performance counter events can reflect a wide variety of common software bugs and can be monitored by hardware with almost no overhead. Second, the counter overflow interrupt supported by existing hardware and operating systems provides a natural and effective mechanism to conduct event sampling at user level. Third, the noise and non-determinism in interrupt delivery complements well with statistical processing. We evaluate PBI using 13 real-world concurrency and sequential bugs from representative open-source server, client, and utility programs, and 10 bugs from a widely used software-testing benchmark. Quantitatively, PBI can effectively diagnose failures caused by these bugs with a small overhead that is never higher than 10%. Qualitatively, PBI does not require any change to software and presents a novel use of existing hardware performance counters.

Quantitative evaluation of fault propagation in a commercial cloud system

“…The high detector shows a limited performance overhead. 30 The detection results are shown in Figure 5. After adding the high detector, the SDC rate decreased from 20.49% to 4.45%, and shows that the detection effect of the high detector was obviously positive.…”

Low-overhead and fully automated statistical debugging with abstraction refinement

“…In other words, although sampling collects less data from each run at each end-user, to achieve statistical significance, more runs/endusers need to be involved and their data need to be transferred, leading to increased latency for failure diagnosis and delayed patch design. For example, under the common 1/100 or 1/1000 sampling rate, hundreds or thousands more failure runs need to be traced before sufficient predicates get sampled to produce statistically meaningful results [4,19,23]. Furthermore, a whole-program sampling infrastructure may lead to a large baseline overhead (e.g., more than 50%) that cannot be amortized through sampling [6].…”

“…The outcomes of these predicates are obtained through software instrumentation or hardware support [4], and constitute the profile of each run. Finally, a profile consists of a set of predicate counts, each recording the number of times a predicate is observed true during the run.…”

“…As an iterative approach, our technique needs to run the multi-iteration refinement process. Like all sampling-based debugging or iterative debugging techniques [4,7,19,23], our technique reduces the user-side overhead of running a heavily instrumented program, but increases the latency of collecting debugging information. This trade-off is widely considered to be worthwhile by previous work, considering the stringent user-side overhead requirement.…”

“…The cooperative statistical debugging approach was first proposed in the Cooperative Bug Isolation (CBI) work [24,25], and has been well researched since then [4,5,7,10,19,35]. Its key idea is to collect execution information from both failing and passing runs of production-run software at many end users' sites, and apply statistical techniques to analyze the collected traces and identify likely failure root causes.…”