Empirical study on the discrepancy between performance testing results from virtual and physical environments

Arif, Muhammad Moiz; Shang, Weiyi; Shihab, Emad

doi:10.1007/s10664-017-9553-x

Cited by 17 publications

(16 citation statements)

References 53 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Source code is only one of the many factors influencing performance variability; others are, to name a few, dynamic compiler optimizations, memory layout, environment variables, virtualization, and OS-dependant factors (Georges et al 2007;Mytkowicz et al 2009;Curtsinger and Berger 2013;de Oliveira et al 2013;Arif et al 2018;Maricq et al 2018;Laaber et al 2019). These other factors could potentially improve the prediction performance of our model even further.…”

Section: Execution Environment Proxiesmentioning

confidence: 98%

“…The variability can be due to co-located tenants, hardware, OS specifics, or source code (Maricq et al 2018;Laaber et al 2019). In particular, virtualized (Arif et al 2018) and cloud (Iosup et al 2011;Gillam et al 2013;Leitner and Cito 2016) environments suffer from performance variability when used as performance execution environment.…”

Section: Performance Variabilitymentioning

confidence: 99%

See 1 more Smart Citation

Predicting unstable software benchmarks using static source code features

2021

View full text Add to dashboard Cite

Software benchmarks are only as good as the performance measurements they yield. Unstable benchmarks show high variability among repeated measurements, which causes uncertainty about the actual performance and complicates reliable change assessment. However, if a benchmark is stable or unstable only becomes evident after it has been executed and its results are available. In this paper, we introduce a machine-learning-based approach to predict a benchmark’s stability without having to execute it. Our approach relies on 58 statically-computed source code features, extracted for benchmark code and code called by a benchmark, related to (1) meta information, e.g., lines of code (LOC), (2) programming language elements, e.g., conditionals or loops, and (3) potentially performance-impacting standard library calls, e.g., file and network input/output (I/O). To assess our approach’s effectiveness, we perform a large-scale experiment on 4,461 Go benchmarks coming from 230 open-source software (OSS) projects. First, we assess the prediction performance of our machine learning models using 11 binary classification algorithms. We find that Random Forest performs best with good prediction performance from 0.79 to 0.90, and 0.43 to 0.68, in terms of AUC and MCC, respectively. Second, we perform feature importance analyses for individual features and feature categories. We find that 7 features related to meta-information, slice usage, nested loops, and synchronization application programming interfaces (APIs) are individually important for good predictions; and that the combination of all features of the called source code is paramount for our model, while the combination of features of the benchmark itself is less important. Our results show that although benchmark stability is affected by more than just the source code, we can effectively utilize machine learning models to predict whether a benchmark will be stable or not ahead of execution. This enables spending precious testing time on reliable benchmarks, supporting developers to identify unstable benchmarks during development, allowing unstable benchmarks to be repeated more often, estimating stability in scenarios where repeated benchmark execution is infeasible or impossible, and warning developers if new benchmarks or existing benchmarks executed in new environments will be unstable.

show abstract

Section: Execution Environment Proxiesmentioning

confidence: 98%

Section: Performance Variabilitymentioning

confidence: 99%

Predicting unstable software benchmarks using static source code features

2021

View full text Add to dashboard Cite

show abstract

“…Nevertheless, all of these studies expect an as-stable-aspossible environment to run performance experiments on. More recently, Arif et al [2] studied the effect of virtual environments on load tests, showing a discrepancy between physical and virtual environments, and Wang et al [33] proposed an approach to assess whether running performance tests in the cloud meets certain goals.…”

Section: Related Workmentioning

confidence: 99%

Continuous software performance assessment: detecting performance problems of software libraries on every build

Laaber

2019

Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis

View full text Add to dashboard Cite

Degradation of software performance can become costly for companies and developers, yet it is hardly assessed continuously. A strategy that would allow continuous performance assessment of software libraries is software microbenchmarking, which faces problems such as excessive execution times and unreliable results that hinder widespread adoption in continuous integration. In my research, I want to develop techniques that allow including software microbenchmarks into continuous integration by utilizing cloud infrastructure and execution time reduction techniques. These will allow assessing performance on every build and therefore catching performance problems before they are released into the wild.

show abstract

“…All of these studies expect an as stable as possible environment to run performance experiments on. More recently, Arif et al [2] study the effect virtual environments have on load tests. They find that there is a discrepancy between physical and virtual environments which are most strongly affected by unpredictability of IO performance.…”

Section: Related Workmentioning

confidence: 99%

“…They may wish to evaluate the performance of applications under "realistic conditions", which nowadays often means running it in the cloud. Finally, they may wish to make use of the myriad of industrial-strength infrastructure automation tools, such as Chef 1 or AWS CloudFormation 2 , which ease the setup and identical repetition of complex performance experiments.…”

Section: Introductionmentioning

confidence: 99%

Performance testing in the cloud. How bad is it really?

Laaber

Scheuner

Leitner

2018

Preprint

View full text Add to dashboard Cite

Rigorous performance engineering traditionally assumes measuring on bare-metal environments to control for as many confounding factors as possible. Unfortunately, some researchers and practitioners might not have access, knowledge, or funds to operate dedicated performance testing hardware, making public clouds an attractive alternative. However, cloud environments are inherently unpredictable and variable with respect to their performance. In this study, we explore the effects of cloud environments on the variability of performance testing outcomes, and to what extent regressions can still be reliably detected. We focus on software microbenchmarks as an example of performance tests, and execute extensive experiments on three different cloud services (AWS, GCE, and Azure) and for different types of instances. We also compare the results to a hosted bare-metal offering from IBM Bluemix. In total, we gathered more than 5 million unique microbenchmarking data points from benchmarks written in Java and Go. We find that the variability of results differs substantially between benchmarks and instance types (from 0.03% to > 100% relative standard deviation). We also observe that testing using Wilcoxon rank-sum generally leads to unsatisfying results for detecting regressions due to a very high number of false positives in all tested configurations. However, simply testing for a difference in medians can be employed with good success to detect even small differences. In some cases, a difference as low as a 1% shift in median execution time can be found with a low false positive rate given a large sample size of 20 instances.

show abstract

Empirical study on the discrepancy between performance testing results from virtual and physical environments

Cited by 17 publications

References 53 publications

Predicting unstable software benchmarks using static source code features

Predicting unstable software benchmarks using static source code features

Continuous software performance assessment: detecting performance problems of software libraries on every build

Performance testing in the cloud. How bad is it really?

Contact Info

Product

Resources

About