Perphecy: Performance Regression Test Selection Made Simple but Effective

IIEEE Trans. Software Eng.

Panichella

et al. 2021

Automated test case generation is an effective technique to yield high-coverage test suites. While the majority of research effort has been devoted to satisfying coverage criteria, a recent trend emerged towards optimizing other non-coverage aspects. In this regard, runtime and memory usage are two essential dimensions: less expensive tests reduce the resource demands for the generation process and for later regression testing phases. This study shows that performance-aware test case generation requires solving two main challenges: providing accurate measurements of resource usage with minimal overhead and avoiding detrimental effects on both final coverage and fault detection effectiveness. To tackle these challenges we conceived a set of performance proxies (inspired by previous work on performance testing) that provide an approximation of the test execution costs (i.e., runtime and memory usage). Thus, we propose an adaptive strategy, called pDynaMOSA, which leverages these proxies by extending DynaMOSA, a state-of-the-art evolutionary algorithm in unit testing. Our empirical study -involving 110 non-trivial Java classes-reveals that our adaptive approach has comparable results to DynaMOSA over seven different coverage criteria (including branch, line, and weak mutation coverage) and similar fault detection effectiveness (measured via strong mutation coverage). Additionally, we observe statistically significant improvements regarding runtime and memory usage for test suites with a similar level of target coverage. Our quantitative and qualitative analyses highlight that our adaptive approach facilitates selecting better test inputs, which is an essential factor to test production code with fewer resources.

Section: Performance Proxiessupporting

confidence: 68%

Section: Performance Proxiesmentioning

confidence: 99%

Testing with Fewer Resources: An Adaptive Approach to Performance-Aware Test Case Generation

Grano

IIEEE Trans. Software Eng.

Panichella

et al. 2021

“…SMBs are different from unit tests, e.g., they test the "usual" path of a program rather then the exceptional, are highly parameterized which results in multiple SMBs having the same call path, and have a result distribution compared to a binary outcome. Research has focused on whether a commit should be performance tested at all [13,28], or it employed performance impact prediction as a driver for prioritization and selection [5,23]. It is unclear though how traditional TCP/RTS techniques perform for SMBs and whether performance impact is the only important goal to optimize for.…”

Section: Rq 3: Reducing Benchmark Execution Timementioning

confidence: 99%

“…Performance test regression selection and prioritization research so far explored testing only performance-critical commits [13], or focused on particular types such as collection-intensive software [23] and concurrent classes [25]. De Oliveira et al [5] propose selection of individual benchmarks based on static and dynamic data that assess whether a code change affects the performance of each benchmark. In my research, I want to explore other properties that are essential in performance test prioritization and selection such as importance and and result quality.…”

Section: Related Workmentioning

confidence: 99%

Continuous software performance assessment: detecting performance problems of software libraries on every build

Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis

2019

Degradation of software performance can become costly for companies and developers, yet it is hardly assessed continuously. A strategy that would allow continuous performance assessment of software libraries is software microbenchmarking, which faces problems such as excessive execution times and unreliable results that hinder widespread adoption in continuous integration. In my research, I want to develop techniques that allow including software microbenchmarks into continuous integration by utilizing cloud infrastructure and execution time reduction techniques. These will allow assessing performance on every build and therefore catching performance problems before they are released into the wild.

“…Performance test regression selection research so far explored testing only performance-critical commits [2,20], or focused on particular types such as collectionintensive software [36] and concurrent classes [40]. [10] propose selection of individual benchmarks based on static and dynamic data that assess whether a code change aects the performance of each benchmark. [7] tackle performance-regression testing through stochastic performance logic (SPL) which lets developers describe performance assertions in hypothesis-test-style logical equations.…”

Section: Related Workmentioning

confidence: 99%

An evaluation of open-source software microbenchmark suites for continuous performance assessment

Proceedings of the 15th International Conference on Mining Software Repositories

Leitner

2018

Continuous integration (CI) emphasizes quick feedback to developers. This is at odds with current practice of performance testing, which predominantely focuses on long-running tests against entire systems in production-like environments. Alternatively, software microbenchmarking attempts to establish a performance baseline for small code fragments in short time. This paper investigates the quality of microbenchmark suites with a focus on suitability to deliver quick performance feedback and CI integration. We study ten open-source libraries written in Java and Go with benchmark suite sizes ranging from 16 to 983 tests, and runtimes between 11 minutes and 8.75 hours. We show that our study subjects include benchmarks with result variability of 50% or higher, indicating that not all benchmarks are useful for reliable discovery of slowdowns. We further arti cially inject actual slowdowns into public API methods of the study subjects and test whether test suites are able to discover them. We introduce a performance-test quality metric called the API benchmarking score (ABS). ABS represents a benchmark suite's ability to nd slowdowns among a set of de ned core API methods. Resulting benchmarking scores (i.e., fraction of discovered slowdowns) vary between 10% and 100% for the study subjects. This paper's methodology and results can be used to (1) assess the quality of existing microbenchmark suites, (2) select a set of tests to be run as part of CI, and (3) suggest or generate benchmarks for currently untested parts of an API.