Dynamically reconfiguring software microbenchmarks: reducing execution time without sacrificing result quality

Laaber, Christoph; Würsten, Stefan; Gall, Harald; Leitner, Philipp

doi:10.1145/3368089.3409683

Cited by 16 publications

(28 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent work studied how and to which degree such an unreliable environment can be used ( Laaber, Scheuner & Leitner, 2019 ; Bulej et al, 2020 ). Similar to He et al (2019) but for microbenchmarks, Laaber et al (2020) introduced dynamic reconfiguration to stop the execution when their result is stable in order to reduce execution time. Our approach to remove redundancies is an alternative approach to reduce microbenchmark suite execution time.…”

Section: Related Workmentioning

confidence: 99%

Using application benchmark call graphs to quantify and improve the practical relevance of microbenchmark suites

Grambow

Laaber

Leitner

et al. 2021

PeerJ Computer Science

Self Cite

View full text Add to dashboard Cite

Performance problems in applications should ideally be detected as soon as they occur, i.e., directly when the causing code modification is added to the code repository. To this end, complex and cost-intensive application benchmarks or lightweight but less relevant microbenchmarks can be added to existing build pipelines to ensure performance goals. In this paper, we show how the practical relevance of microbenchmark suites can be improved and verified based on the application flow during an application benchmark run. We propose an approach to determine the overlap of common function calls between application and microbenchmarks, describe a method which identifies redundant microbenchmarks, and present a recommendation algorithm which reveals relevant functions that are not covered by microbenchmarks yet. A microbenchmark suite optimized in this way can easily test all functions determined to be relevant by application benchmarks after every code change, thus, significantly reducing the risk of undetected performance problems. Our evaluation using two time series databases shows that, depending on the specific application scenario, application benchmarks cover different functions of the system under test. Their respective microbenchmark suites cover between 35.62% and 66.29% of the functions called during the application benchmark, offering substantial room for improvement. Through two use cases—removing redundancies in the microbenchmark suite and recommendation of yet uncovered functions—we decrease the total number of microbenchmarks and increase the practical relevance of both suites. Removing redundancies can significantly reduce the number of microbenchmarks (and thus the execution time as well) to ~10% and ~23% of the original microbenchmark suites, whereas recommendation identifies up to 26 and 14 newly, uncovered functions to benchmark to improve the relevance. By utilizing the differences and synergies of application benchmarks and microbenchmarks, our approach potentially enables effective software performance assurance with performance tests of multiple granularities.

show abstract

Section: Related Workmentioning

confidence: 99%

Using application benchmark call graphs to quantify and improve the practical relevance of microbenchmark suites

Grambow

Laaber

Leitner

et al. 2021

PeerJ Computer Science

Self Cite

View full text Add to dashboard Cite

show abstract

“…Therefore, we aim to select a diverse set of projects from different domains, with varying benchmark suite sizes, and a multitude of versions to apply TCP on. To this end, we perform purposive sampling (Baltes and Ralph 2020) of Github projects based on a list of 1,545 projects with JMH suites from Laaber et al (2020).…”

Section: Study Objectsmentioning

confidence: 99%

Applying test case prioritization to software microbenchmarks

2021

Self Cite

View full text Add to dashboard Cite

Regression testing comprises techniques which are applied during software evolution to uncover faults effectively and efficiently. While regression testing is widely studied for functional tests, performance regression testing, e.g., with software microbenchmarks, is hardly investigated. Applying test case prioritization (TCP), a regression testing technique, to software microbenchmarks may help capturing large performance regressions sooner upon new versions. This may especially be beneficial for microbenchmark suites, because they take considerably longer to execute than unit test suites. However, it is unclear whether traditional unit testing TCP techniques work equally well for software microbenchmarks. In this paper, we empirically study coverage-based TCP techniques, employing total and additional greedy strategies, applied to software microbenchmarks along multiple parameterization dimensions, leading to 54 unique technique instantiations. We find that TCP techniques have a mean APFD-P (average percentage of fault-detection on performance) effectiveness between 0.54 and 0.71 and are able to capture the three largest performance changes after executing 29% to 66% of the whole microbenchmark suite. Our efficiency analysis reveals that the runtime overhead of TCP varies considerably depending on the exact parameterization. The most effective technique has an overhead of 11% of the total microbenchmark suite execution time, making TCP a viable option for performance regression testing. The results demonstrate that the total strategy is superior to the additional strategy. Finally, dynamic-coverage techniques should be favored over static-coverage techniques due to their acceptable analysis overhead; however, in settings where the time for prioritzation is limited, static-coverage techniques provide an attractive alternative.

show abstract

“…In Section 2, we describe this repetition and its impact. However, finding the right configuration is nontrivial and developers often get it "wrong", as shown by Laaber et al (2020). Their solution is to dynamically-during the execution of a benchmark-decide when to stop a benchmark; still, this dynamic reconfiguration requires manually setting upper bounds on the iteration parameter.…”

Section: Application Scenariosmentioning

confidence: 99%

“…This would enable developers to take extra care of these benchmarks and setting the iteration configuration to higher values than set by default or proposed by research. If dynamic reconfiguration by Laaber et al (2020) is used, the upper bounds for the number of iterations can be pessimistically set to high values for unstable benchmarks.…”

Section: Application Scenariosmentioning

confidence: 99%

See 1 more Smart Citation

Predicting unstable software benchmarks using static source code features

2021

Self Cite

View full text Add to dashboard Cite

Software benchmarks are only as good as the performance measurements they yield. Unstable benchmarks show high variability among repeated measurements, which causes uncertainty about the actual performance and complicates reliable change assessment. However, if a benchmark is stable or unstable only becomes evident after it has been executed and its results are available. In this paper, we introduce a machine-learning-based approach to predict a benchmark’s stability without having to execute it. Our approach relies on 58 statically-computed source code features, extracted for benchmark code and code called by a benchmark, related to (1) meta information, e.g., lines of code (LOC), (2) programming language elements, e.g., conditionals or loops, and (3) potentially performance-impacting standard library calls, e.g., file and network input/output (I/O). To assess our approach’s effectiveness, we perform a large-scale experiment on 4,461 Go benchmarks coming from 230 open-source software (OSS) projects. First, we assess the prediction performance of our machine learning models using 11 binary classification algorithms. We find that Random Forest performs best with good prediction performance from 0.79 to 0.90, and 0.43 to 0.68, in terms of AUC and MCC, respectively. Second, we perform feature importance analyses for individual features and feature categories. We find that 7 features related to meta-information, slice usage, nested loops, and synchronization application programming interfaces (APIs) are individually important for good predictions; and that the combination of all features of the called source code is paramount for our model, while the combination of features of the benchmark itself is less important. Our results show that although benchmark stability is affected by more than just the source code, we can effectively utilize machine learning models to predict whether a benchmark will be stable or not ahead of execution. This enables spending precious testing time on reliable benchmarks, supporting developers to identify unstable benchmarks during development, allowing unstable benchmarks to be repeated more often, estimating stability in scenarios where repeated benchmark execution is infeasible or impossible, and warning developers if new benchmarks or existing benchmarks executed in new environments will be unstable.

show abstract

Dynamically reconfiguring software microbenchmarks: reducing execution time without sacrificing result quality

Cited by 16 publications

References 42 publications

Using application benchmark call graphs to quantify and improve the practical relevance of microbenchmark suites

Using application benchmark call graphs to quantify and improve the practical relevance of microbenchmark suites

Applying test case prioritization to software microbenchmarks

Predicting unstable software benchmarks using static source code features

Contact Info

Product

Resources

About