Identifying performance deviations in thread pools

Abstract

Large-scale software systems handle increasingly larger workloads by implementing highly concurrent and dis tributed design patterns. The thread pool pattern uses pools of pre-existing and reusable threads to limit thread lifecycle over head (thread creation and destruction) and resource thrashing (thread proliferation). However, these advantages are weighed against performance issues caused by concurrency risks, like synchronization errors or deadlock, and thread pool-speclfic risks, like poorly tuned pool size or thread leakage. Detecting these performance issues during load testing requires a thorough understanding of how thread pools behave, yet most performance analysts have limited knowledge of the system and are Hooded with terabytes of data from load tests. We propose a methodology to identify threads with performance deviations in thread pools.Our methodology ranks threads based on the dissimil arity of their resource usage metries. A case study on a large-scale in dustrial software system shows that our methodology can identify threads with performance deviations with an average precision of 100% and an average recall of 76.61%. Our methodology performs very well when ranking long-lived deviations, such as memory leaks, but more work is needed to rank short-lived deviations, such as CPU spikes.Keywords-thread-pools; behaviour-based clustering; under standing ULS systems; I. IN TRODUCTIONThe rise of ultra-large-scale (ULS) e-commerce and telecommunication systems, like Amazon or Facebook, poses a new challenge for the software maintenance field. ULS sys tems require near-perfect up-time and potentially must support hundreds or even thousands of concurrent connections and operations. Failures in such systems are more often associated with an inability to scale to performance demands, than with feature bugs [1], [2]. To ensure that ULS systems are able to perform according to the expected workloads, performance analysts have become key players in the development and maintenance of today's en terprises. Performance analysts are responsible for performing load tests on systems and monitoring how the systems behave under realistic workloads. Such load tests allow performance analysts to determine the maximum operating capacity of a system, to validate non-functional requirements and to uncover bottlenecks. A load test typically lasts from several hours to several days and generates terabytes of performance data and execution logs [3]. Despite the crucial role of performance analysts, current research tends to ignore their needs. The current state of the practice requires significant manual review of the performance data and execution logs and a high degree of insight into the 978-1-4577-0664-6/1lI$26.00 ©2011 IEEE 83 system behaviour [2], [4], [5]. Tool support is lacking because most research techniques require either heavy instrumentation (disrupting the work-load) or more detailed log entries [2], [5]-[8] (often unavailable during load tests because collecting detailed logs impacts the system's performance...

Detecting Discontinuities in Large Scale Systems

“…Memory Stress: According to BlackBerry and Mozilla, the most common anomaly occurring in the field is related to Transient memory issues [30][31]. Transient memory issues (memory spikes) are large increases in memory usage over a relatively short period of time.…”

“…Syers et al [30,31] proposed an approach to identify performance anomalies and deviation in thread pool using performance counters. Their approach is limited to the detection memory related performance anomalies in enterprise systems (e.g., memory leaks, memory spikes and memory blots).…”

“…[32][33][34]. Finally, there are other approaches [27,29] that use annotated software models to detect performance anomalies [30]. These approaches, however, use software model simulations and not real production software.…”

Performance optimization of deployed software-as-a-service applications

“…Syer et al [35] use covariance matrices to detect deviations in thread pools that indicate possible performance problems. The focus of their approach is on thread pools while ours is not limited to a particular architectural pattern.…”

Locating performance improvement opportunities in an industrial software-as-a-service application

“…Syer et al [15] use covariance matrices to detect deviations in thread pools that indicate possible performance problems. The focus of their approach is on thread pools while ours is not limited to a particular architectural pattern.…”