Towards reducing the time needed for load testing

Abstract

The performance of large‐scale systems must be thoroughly tested under various levels of workload, as load‐related issues can have a disastrous impact on the system. However, load testing often requires a large amount of time, running from hours to even days. In our prior work, we reduced the execution time of a load test by detecting repetitiveness in individual performance metric values, such as CPU utilization, that are observed during the test. However, as we explain in this paper, disregarding combinations of performance metrics may miss important information about the load‐related behavior of a system. In this paper we revisit our prior approach, by proposing an approach that reduces the execution time of a load test by detecting whether a test no longer exercises new combinations of the observed performance metrics. We study three open source systems, in which we use our new and prior approaches to reduce the execution time of a 24‐hour load test. We show that our new approach is capable of reducing the execution time of the test to less than 8.5 hours, while preserving a coverage of at least 95% of the combinations that are observed between the performance metrics during the 24‐hour tests.

Dynamically reconfiguring software microbenchmarks: reducing execution time without sacrificing result quality

“…JMH 1 is the de-facto standard framework for writing and executing software microbenchmarks (in the following simply called benchmarks) for Java. Benchmarks operate on the same level of granularity as unit tests, i.e., statement/method level, and are similarly defined in code and configured through annotations.…”

“…To lower the time spent in performance testing activities, previous research applied techniques to select which commits to test [24,45] or which tests to run [3,14], to prioritize tests that are more likely to expose slowdowns [39], and to stop load tests once they become repetitive [1,2] or do not improve result accuracy [20]. However, none of these approaches are tailored to and consider characteristics of software microbenchmarks and enable running full benchmark suites, reduce the overall runtime, while still maintaining the same result quality.…”

“…Our empirical evaluation comprises of three different stoppage criteria, including the one from He et al [20]. It assesses whether benchmarks executed with dynamic reconfiguration in controlled, bare-metal environments (1) maintain their result quality and (2) have shorter execution times, compared to being executed with the default JMH configuration.…”

Using application benchmark call graphs to quantify and improve the practical relevance of microbenchmark suites

“…Other approaches aim to reduce the execution time for application benchmarks: AlGhamdi et al (2016 , 2020 ) proposed to stop the benchmark run when the system reaches a repetitive performance state, and He et al (2019) devised a statistical approach based on kernel density estimation to stop once a benchmark is unlikely to produce a different result with more repetitions. Such approaches can only be combined with our analysis and optimization under certain conditions.…”

Predicting unstable software benchmarks using static source code features

“…Traditionally, research on performance testing focussed mostly on load testing, such as identifying problems and reporting on case studies (Weyuker and Vokolos 2000;Menascé 2002;Jiang and Hassan 2015). More recent work focussed on industrial applicability (Nguyen et al 2014;Foo et al 2015;Chen et al 2019) and reducing the time spent in load testing activities (AlGhamdi et al 2016;AlGhamdi et al 2020;He et al 2019).…”