Applying test case prioritization to software microbenchmarks

Abstract

Regression testing comprises techniques which are applied during software evolution to uncover faults effectively and efficiently. While regression testing is widely studied for functional tests, performance regression testing, e.g., with software microbenchmarks, is hardly investigated. Applying test case prioritization (TCP), a regression testing technique, to software microbenchmarks may help capturing large performance regressions sooner upon new versions. This may especially be beneficial for microbenchmark suites, because they take considerably longer to execute than unit test suites. However, it is unclear whether traditional unit testing TCP techniques work equally well for software microbenchmarks. In this paper, we empirically study coverage-based TCP techniques, employing total and additional greedy strategies, applied to software microbenchmarks along multiple parameterization dimensions, leading to 54 unique technique instantiations. We find that TCP techniques have a mean APFD-P (average percentage of fault-detection on performance) effectiveness between 0.54 and 0.71 and are able to capture the three largest performance changes after executing 29% to 66% of the whole microbenchmark suite. Our efficiency analysis reveals that the runtime overhead of TCP varies considerably depending on the exact parameterization. The most effective technique has an overhead of 11% of the total microbenchmark suite execution time, making TCP a viable option for performance regression testing. The results demonstrate that the total strategy is superior to the additional strategy. Finally, dynamic-coverage techniques should be favored over static-coverage techniques due to their acceptable analysis overhead; however, in settings where the time for prioritzation is limited, static-coverage techniques provide an attractive alternative.

The Early Microbenchmark Catches the Bug -- Studying Performance Issues Using Micro- and Application Benchmarks

“…Conversely, should 𝑣 2 execute faster than 𝑣 1 , we call the change a performance improvement. A similar technique has been used by Laaber et al for microbenchmarks in the past [26].…”

“…There are alternative metrics and techniques for quantifying performance changes which are likely to give different results. Generally, the combination we used is in line with the state of the art and best practices in benchmarking: Bootstrap confidence intervals are state of the art and have often been used by practitioners of performance engineering [25,26,28], while the median ratio is inspired by the similar mean ratio used by Laaber et al for quantifying performance changes [26]. Furthermore, we preprocess the experiment data from the application benchmark by applying a median operator to each second of the experiment.…”

“…In our study, we always executed the complete microbenchmark suite without further optimizations and thus had a suite execution time of about one hour. The suite execution, however, can be optimized by dynamically stopping microbenchmarks when there are stable results [21], adjust the execution order of microbenchmarks within the suite [26,29,31], or selecting and execution only a subset of relevant microbenchmarks [11,17]. Our testbed application with a microbenchmark suite covering almost all source code parts can also be used to evaluate these optimization approaches.…”

An exploratory study of history-based test case prioritization techniques on different datasets

“…Execution time 10 will be used to measure the performance of the technique in terms of overhead incurred. APFD has been used in studies [33][34][35][36] to measure the effectiveness of TCP techniques. APFD can be calculated by the formula given in Eq.…”

Towards effective assessment of steady state performance in Java software: are we there yet?

“…In fact, different developers can potentially execute the same microbenchmark on a wide range of different machines/environments, given the inherently distributed nature of open-source software development. On top of that, there is an increasing interest in promoting the adoption of microbenchmarks in CI (Laaber and Leitner 2018;Laaber et al 2019Laaber et al , 2020Laaber et al , 2021, which are most likely uncontrolled/noisy environments. All these aspects make it impractical to identify a reference environment for each benchmark/system.…”