Automatic Comparison of Load Tests to Support the Performance Analysis of Large Enterprise Systems

Abstract

Load testing is crucial to uncover functional and performance bugs in large-scale systems. Load tests generate vast amounts of performance data, which needs to be compared and analyzed in limited time across tests. This helps performance analysts to understand the resource usage of an application and to find out if an application is meeting its performance goals. The biggest challenge for performance analysts is to identify the few important performance counters in the highly redundant performance data. In this paper, we employed a statistical technique, Principal Component Analysis (PCA) to reduce the large volume of performance counter data, to a smaller, more meaningful and manageable set. Furthermore, our methodology automates the process of comparing the important counters across load tests to identify performance gains/losses. A case study on load test data of a large enterprise application shows that our methodology can effectively guide performance analysts to identify and compare top performance counters across tests in limited time.

Detecting Discontinuities in Large Scale Systems

“…In this step, analysts sanitize and preprocess the data to make it suitable for the forecasting techniques selected in the previous step. During sanitation missing, ignorable, erroneous and empty performance counter variables are treated [2][3][4]. Counter data is missing when a performance monitor fails to record an instance of a performance counter.…”

“…We provide an overview of the PCA based performance counter selection technique in this paper. Further details are discussed in our previous work [7]. Basically, the high level goal of using PCA in our context is the same as using clustering: selecting the least correlated subset of performance counters that can still explain the maximum variations in the data, thereby eliminating performance counters capturing little variance such as invariants and configurations related performance counters.…”

“…Except for experiment 5, which consisted of production data obtained from the industrial partner and experiment 4, which consisted of synthesizing data using mathematical equations, all other performance test experiments are repeated > 30 times (as suggested by Georges et al [19]) to ensure that measure of variation is not misleading or incorrect, to overcome performance counters instability during the experiments, and to ensure consistency among our findings. The ramp-up and ramp-down (warm up and cool down) [5,7] periods, usually spanning over 15 minutes were excluded from our analysis, as the system is usually not stable during these periods during performance tests. We used windows 'perfmon' [20] tool to collect the performance data after every 15 seconds (sampling interval) across all the eight machines.…”

Performance optimization of deployed software-as-a-service applications

“…Malik et al [36] have presented an approach for narrowing down the set of performance counters that have to be monitored to automatically compare load tests by using statistics. Their technique also ranks the performance counters based on their importance for load tests.…”

Locating performance improvement opportunities in an industrial software-as-a-service application

“…Malik et al [16] have presented an approach for narrowing down the set of performance counters that have to be monitored to automatically compare load tests by using statistics. Their technique also ranks the performance counters based on their importance for load tests.…”