Conducting Repeatable Experiments in Highly Variable Cloud Computing Environments

Abedi, Ali; Brecht, Tim

doi:10.1145/3030207.3030229

Cited by 38 publications

(47 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For each cloud instance type as listed in Table 2, we create 50 different instances. On each instance, we schedule 10 experiment trials of each benchmark in randomized order (following the method proposed by Abedi and Brecht [1]) without breaks between trials. Within each trial, every benchmark (e.g., etcd-1) consists of 50 repeated executions (e.g., using the -i50 parameter of JMH) and every execution produces a single data point, which reports the average execution time in ns.…”

Section: Packagementioning

confidence: 99%

“…Time [ns] Instance Variability − etcd−5 on GCE Mem The second group we discuss exhibits high variability on some, but not all, instance types. This group contains two sub-types, (1) benchmarks that have high standard deviations, but where the median runtime is similar, and (2) benchmarks that have overall varying results, including substantially differing medians on different instances. An example for the first sub-group is log4j2-3 on GCE Mem -and similarly on the other GCE and Azure instances -where the benchmark's variability differs among the instances of the same instance types (see Figure 3).…”

Section: Variability In the Cloudmentioning

confidence: 99%

“…We now randomly select 1, 5, 10, and 20 instances (as indicated by the column "# VMs") and then randomly select 5 trials for the test and 5 trials for the control group. This implements a recent best practice for low cloud measurement variabilities as proposed by Abedi and Brecht [1].…”

Section: A/a Testing We Initially Study the Applicability Ofmentioning

confidence: 99%

“…They may wish to evaluate the performance of applications under "realistic conditions", which nowadays often means running it in the cloud. Finally, they may wish to make use of the myriad of industrial-strength infrastructure automation tools, such as Chef 1 or AWS CloudFormation 2 , which ease the setup and identical repetition of complex performance experiments.…”

Section: Introductionmentioning

confidence: 99%

“…Further, we study instances hosted in three of the most prominent public IaaS providers (Google Compute Engine, AWS EC2, and Microsoft Azure) and contrast the results against a dedicated bare-metal machine deployed using IBM Bluemix (formerly Softlayer). We also evaluate and compare the impact of common deployment strategies for performance tests in the cloud, namely running test and control experiments on different cloud instances, on the same instances, and randomized multiple interleaved trials as recently proposed as best practice [1].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Performance testing in the cloud. How bad is it really?

Laaber

Scheuner

Leitner

2018

Preprint

View full text Add to dashboard Cite

Rigorous performance engineering traditionally assumes measuring on bare-metal environments to control for as many confounding factors as possible. Unfortunately, some researchers and practitioners might not have access, knowledge, or funds to operate dedicated performance testing hardware, making public clouds an attractive alternative. However, cloud environments are inherently unpredictable and variable with respect to their performance. In this study, we explore the effects of cloud environments on the variability of performance testing outcomes, and to what extent regressions can still be reliably detected. We focus on software microbenchmarks as an example of performance tests, and execute extensive experiments on three different cloud services (AWS, GCE, and Azure) and for different types of instances. We also compare the results to a hosted bare-metal offering from IBM Bluemix. In total, we gathered more than 5 million unique microbenchmarking data points from benchmarks written in Java and Go. We find that the variability of results differs substantially between benchmarks and instance types (from 0.03% to > 100% relative standard deviation). We also observe that testing using Wilcoxon rank-sum generally leads to unsatisfying results for detecting regressions due to a very high number of false positives in all tested configurations. However, simply testing for a difference in medians can be employed with good success to detect even small differences. In some cases, a difference as low as a 1% shift in median execution time can be found with a low false positive rate given a large sample size of 20 instances.

show abstract

Section: Packagementioning

confidence: 99%

Section: Variability In the Cloudmentioning

confidence: 99%

Section: A/a Testing We Initially Study the Applicability Ofmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Performance testing in the cloud. How bad is it really?

Laaber

Scheuner

Leitner

2018

Preprint

View full text Add to dashboard Cite

show abstract

A multi‐faceted analysis of the performance variability of virtual machines

et al. 2023

View full text Add to dashboard Cite

Cloud computing and virtualization solutions allow one to rent the virtual machines (VMs) needed to run applications on a pay‐per‐use basis, but rented VMs do not offer any guarantee on their performance. Cloud platforms are known to be affected by performance variability, but a better understanding is still required. This article moves in that direction and presents an in‐depth, multi‐faceted study on the performance variability of VMs. Unlike previous studies, our assessment covers a wide range of factors: 16 VM types from 4 well‐known cloud providers, 10 benchmarks, and 28 different metrics. We present four new contributions. First, we introduce a new benchmark suite (VMBS) that let researchers and practitioners systematically collect a diverse set of performance data. Second, we present a new indicator, called VI, that allows for measuring variability in the performance of VMs. Third, we illustrate an analysis of the collected data across four different dimensions: resources, isolation, time, and cost. Fourth, we present multiple predictive models based on machine learning (ML) that aim to forecast future performance and detect time patterns. Our experiments provide important insights on the resource variability of VMs, highlighting differences and similarities between various cloud providers. To the best of our knowledge, this is the widest analysis ever conducted on the topic.

show abstract

Software microbenchmarking in the cloud. How bad is it really?

2019

View full text Add to dashboard Cite

Rigorous performance engineering traditionally assumes measuring on bare-metal environments to control for as many confounding factors as possible. Unfortunately, some researchers and practitioners might not have access, knowledge, or funds to operate dedicated performance-testing hardware, making public clouds an attractive alternative. However, shared public cloud environments are inherently unpredictable in terms of the system performance they provide. In this study, we explore the effects of cloud environments on the variability of performance test results and to what extent slowdowns can still be reliably detected even in a public cloud. We focus on software microbenchmarks as an example of performance tests and execute extensive experiments on three different well-known public cloud services (AWS, GCE, and Azure) using three different cloud instance types per service. We also compare the results to a hosted bare-metal offering from IBM Bluemix. In total, we gathered more than 4.5 million unique microbenchmarking data points from benchmarks written in Java and Go. We find that the variability of results differs substantially between benchmarks and instance types (by a coefficient of variation from 0.03% to > 100%). However, executing test and control ex

show abstract

Conducting Repeatable Experiments in Highly Variable Cloud Computing Environments

Cited by 38 publications

References 8 publications

Performance testing in the cloud. How bad is it really?

Performance testing in the cloud. How bad is it really?

A multi‐faceted analysis of the performance variability of virtual machines

Software microbenchmarking in the cloud. How bad is it really?

Contact Info

Product

Resources

About