The Truth, The Whole Truth, and Nothing But the Truth

Blackburn, Stephen M.; Diwan, Amer; Hauswirth, Matthias; Sweeney, Peter F.; Amaral, José Nelson; Brecht, Tim; Bulej, Lubomír; Click, Cliff; Eeckhout, Lieven; Fischmeister, Sebastian; Frampton, Daniel; Hendren, Laurie; Hind, Michael; Hosking, Antony L.; Jones, Richard; Kalibera, Tomas; Keynes, Nathan; Nystrom, Nathaniel; Zeller, Andreas

doi:10.1145/2983574

Cited by 22 publications

(4 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We execute each benchmark for 3 trials consisting of 20 iterations of 1s duration. This configuration is in line with other recent performance engineering works, e.g., Blackburn et al (2016), Chen et al (2020), andMühlbauer et al (2020). Nonetheless, it does not ensure that the measurements are stable, i.e., measurement variability is low.…”

Section: Internal Validitysupporting

confidence: 66%

Applying test case prioritization to software microbenchmarks

2021

View full text Add to dashboard Cite

Regression testing comprises techniques which are applied during software evolution to uncover faults effectively and efficiently. While regression testing is widely studied for functional tests, performance regression testing, e.g., with software microbenchmarks, is hardly investigated. Applying test case prioritization (TCP), a regression testing technique, to software microbenchmarks may help capturing large performance regressions sooner upon new versions. This may especially be beneficial for microbenchmark suites, because they take considerably longer to execute than unit test suites. However, it is unclear whether traditional unit testing TCP techniques work equally well for software microbenchmarks. In this paper, we empirically study coverage-based TCP techniques, employing total and additional greedy strategies, applied to software microbenchmarks along multiple parameterization dimensions, leading to 54 unique technique instantiations. We find that TCP techniques have a mean APFD-P (average percentage of fault-detection on performance) effectiveness between 0.54 and 0.71 and are able to capture the three largest performance changes after executing 29% to 66% of the whole microbenchmark suite. Our efficiency analysis reveals that the runtime overhead of TCP varies considerably depending on the exact parameterization. The most effective technique has an overhead of 11% of the total microbenchmark suite execution time, making TCP a viable option for performance regression testing. The results demonstrate that the total strategy is superior to the additional strategy. Finally, dynamic-coverage techniques should be favored over static-coverage techniques due to their acceptable analysis overhead; however, in settings where the time for prioritzation is limited, static-coverage techniques provide an attractive alternative.

show abstract

Section: Internal Validitysupporting

confidence: 66%

Applying test case prioritization to software microbenchmarks

2021

View full text Add to dashboard Cite

show abstract

“…Result quality is impaired by not running enough measurements as well as measurement bias, which requires careful experiment planning and execution [7,11,15,18,19,28,40]. To mitigate measurement bias, Georges et al [18] outlined a rigorous methodology how to asses performance of Java programs, which we base our measurement technique on.…”

Section: Related Workmentioning

confidence: 99%

Dynamically reconfiguring software microbenchmarks: reducing execution time without sacrificing result quality

Laaber

Würsten

Gall

et al. 2020

Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Softw

View full text Add to dashboard Cite

“…These values are in line with best practice, i.e., Georges et al (2007) suggest 30 iterations, and cover a range that previous research used for their performance measurements. For example, 5 iterations are used by Blackburn et al (2004), Jangda et al (2019), andMühlbauer et al (2020); 10 iterations are used by Selakovic and Pradel (2016), Song and Lu (2017), and Kaltenecker et al (2019); 20 iterations are used by Laaber and Leitner (2018) and were the default for Java Microbenchmarking Harness (JHM) benchmarks (Shipilev 2018); and 30 iterations are used by Curtsinger and Berger (2013), Blackburn et al (2016), andChen et al (2020).…”

Section: Number Of Benchmark Iterationsmentioning

confidence: 99%

Predicting unstable software benchmarks using static source code features

2021

View full text Add to dashboard Cite

Software benchmarks are only as good as the performance measurements they yield. Unstable benchmarks show high variability among repeated measurements, which causes uncertainty about the actual performance and complicates reliable change assessment. However, if a benchmark is stable or unstable only becomes evident after it has been executed and its results are available. In this paper, we introduce a machine-learning-based approach to predict a benchmark’s stability without having to execute it. Our approach relies on 58 statically-computed source code features, extracted for benchmark code and code called by a benchmark, related to (1) meta information, e.g., lines of code (LOC), (2) programming language elements, e.g., conditionals or loops, and (3) potentially performance-impacting standard library calls, e.g., file and network input/output (I/O). To assess our approach’s effectiveness, we perform a large-scale experiment on 4,461 Go benchmarks coming from 230 open-source software (OSS) projects. First, we assess the prediction performance of our machine learning models using 11 binary classification algorithms. We find that Random Forest performs best with good prediction performance from 0.79 to 0.90, and 0.43 to 0.68, in terms of AUC and MCC, respectively. Second, we perform feature importance analyses for individual features and feature categories. We find that 7 features related to meta-information, slice usage, nested loops, and synchronization application programming interfaces (APIs) are individually important for good predictions; and that the combination of all features of the called source code is paramount for our model, while the combination of features of the benchmark itself is less important. Our results show that although benchmark stability is affected by more than just the source code, we can effectively utilize machine learning models to predict whether a benchmark will be stable or not ahead of execution. This enables spending precious testing time on reliable benchmarks, supporting developers to identify unstable benchmarks during development, allowing unstable benchmarks to be repeated more often, estimating stability in scenarios where repeated benchmark execution is infeasible or impossible, and warning developers if new benchmarks or existing benchmarks executed in new environments will be unstable.

show abstract

The Truth, The Whole Truth, and Nothing But the Truth

Abstract: The version in the Kent Academic Repository may differ from the final published version. Users are advised to check http://kar.kent.ac.uk for the status of the paper. Users should always cite the published version of record.

Cited by 22 publications

References 27 publications

Applying test case prioritization to software microbenchmarks

Applying test case prioritization to software microbenchmarks

Dynamically reconfiguring software microbenchmarks: reducing execution time without sacrificing result quality

Predicting unstable software benchmarks using static source code features

Contact Info

Product

Resources

About