Unit Testing Performance in Java Projects

Stefan, Petr; Horký, Vojtěch; Bulej, Lubomír; Tůma, Petr

doi:10.1145/3030207.3030226

Cited by 35 publications

(39 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Unfortunately, due to a configuration error, we are lacking all results for one benchmark (bleve-1 ) and consequently omit this benchmark from all remaining discussions. We decided to choose Java as it has been ranked highly in programming-language rankings (e.g., Tiobe 6 ), is executed in a VM (i.e., the JVM) with dynamic compiler optimizations, and has a microbenchmarking framework available that is used by real OSS projects (Stefan et al 2017). The Go language complements our study selection as a new programming language being introduced in 2009.…”

Section: Project and Benchmark Selectionmentioning

confidence: 99%

“…IaaS clouds provide relatively low-level access and configurability while allowing for high scalability of experiments; and software microbenchmarks emerge as the performance evaluation strategy of choice for library-type projects. Software microbenchmarks can be seen as the unit-test equivalent for performance and are sometimes even referred to as performance unit tests (Stefan et al 2017). They are relatively short-running (e.g., < 1ms) performance tests against small software units (e.g., methods), which are typically tested in isolation without a fullydeployed system (as used for load testing).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Software microbenchmarking in the cloud. How bad is it really?

2019

View full text Add to dashboard Cite

Rigorous performance engineering traditionally assumes measuring on bare-metal environments to control for as many confounding factors as possible. Unfortunately, some researchers and practitioners might not have access, knowledge, or funds to operate dedicated performance-testing hardware, making public clouds an attractive alternative. However, shared public cloud environments are inherently unpredictable in terms of the system performance they provide. In this study, we explore the effects of cloud environments on the variability of performance test results and to what extent slowdowns can still be reliably detected even in a public cloud. We focus on software microbenchmarks as an example of performance tests and execute extensive experiments on three different well-known public cloud services (AWS, GCE, and Azure) using three different cloud instance types per service. We also compare the results to a hosted bare-metal offering from IBM Bluemix. In total, we gathered more than 4.5 million unique microbenchmarking data points from benchmarks written in Java and Go. We find that the variability of results differs substantially between benchmarks and instance types (by a coefficient of variation from 0.03% to > 100%). However, executing test and control ex

show abstract

Section: Project and Benchmark Selectionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Software microbenchmarking in the cloud. How bad is it really?

2019

View full text Add to dashboard Cite

show abstract

“…In the Java world, the Java Microbenchmarking Harness (JMH) has established itself as the de facto standard to implement software microbenchmarking [24]. JMH is part of the OpenJDK implementation of Java and allows users to specify benchmarks through an annotation mechanism.…”

Section: Software Microbenchmarkingmentioning

confidence: 99%

“…The study is based on 20 microbenchmarks selected from four OSS projects, two of which are written in Java and two in Go. We decided to choose Java as it has been ranked highly in programming language rankings (e.g., Tiobe 5 ), is executed in a virtual machine (VM) (i.e., the JVM) with dynamic compiler optimizations, and has a microbenchmarking framework available that is used by real OSS projects [24]. Go complements our study selection as a new programming language, being introduced in 2009.…”

Section: Packagementioning

confidence: 99%

“…By now, such performance tests are academically well-understood, and recent research focuses on industrial applicability [8,21] or how to reduce the time necessary for load testing [11]. Studies of software microbenchmarking have not received main stream attention previously, but academics have recently started investigating it [5,12,24]. Similarly, Leitner and Bezemer [16] recently investigated different practices of microbenchmarking of OSS written in Java.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Performance testing in the cloud. How bad is it really?

Laaber

Scheuner

Leitner

2018

Preprint

View full text Add to dashboard Cite

Rigorous performance engineering traditionally assumes measuring on bare-metal environments to control for as many confounding factors as possible. Unfortunately, some researchers and practitioners might not have access, knowledge, or funds to operate dedicated performance testing hardware, making public clouds an attractive alternative. However, cloud environments are inherently unpredictable and variable with respect to their performance. In this study, we explore the effects of cloud environments on the variability of performance testing outcomes, and to what extent regressions can still be reliably detected. We focus on software microbenchmarks as an example of performance tests, and execute extensive experiments on three different cloud services (AWS, GCE, and Azure) and for different types of instances. We also compare the results to a hosted bare-metal offering from IBM Bluemix. In total, we gathered more than 5 million unique microbenchmarking data points from benchmarks written in Java and Go. We find that the variability of results differs substantially between benchmarks and instance types (from 0.03% to > 100% relative standard deviation). We also observe that testing using Wilcoxon rank-sum generally leads to unsatisfying results for detecting regressions due to a very high number of false positives in all tested configurations. However, simply testing for a difference in medians can be employed with good success to detect even small differences. In some cases, a difference as low as a 1% shift in median execution time can be found with a low false positive rate given a large sample size of 20 instances.

show abstract