A Survey on Load Testing of Large-Scale Software Systems

Jiang, Zhen Ming; Hassan, Ahmed E.

doi:10.1109/tse.2015.2445340

Cited by 136 publications

(79 citation statements)

References 138 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Traditionally, performance testing research was conducted in the context of system-scale load and stress testing [3,14,19,25]. By now, such performance tests are academically well-understood, and recent research focuses on industrial applicability [8,21] or how to reduce the time necessary for load testing [11].…”

Section: Related Workmentioning

confidence: 99%

Performance testing in the cloud. How bad is it really?

Laaber

Scheuner

Leitner

2018

Preprint

View full text Add to dashboard Cite

Rigorous performance engineering traditionally assumes measuring on bare-metal environments to control for as many confounding factors as possible. Unfortunately, some researchers and practitioners might not have access, knowledge, or funds to operate dedicated performance testing hardware, making public clouds an attractive alternative. However, cloud environments are inherently unpredictable and variable with respect to their performance. In this study, we explore the effects of cloud environments on the variability of performance testing outcomes, and to what extent regressions can still be reliably detected. We focus on software microbenchmarks as an example of performance tests, and execute extensive experiments on three different cloud services (AWS, GCE, and Azure) and for different types of instances. We also compare the results to a hosted bare-metal offering from IBM Bluemix. In total, we gathered more than 5 million unique microbenchmarking data points from benchmarks written in Java and Go. We find that the variability of results differs substantially between benchmarks and instance types (from 0.03% to > 100% relative standard deviation). We also observe that testing using Wilcoxon rank-sum generally leads to unsatisfying results for detecting regressions due to a very high number of false positives in all tested configurations. However, simply testing for a difference in medians can be employed with good success to detect even small differences. In some cases, a difference as low as a 1% shift in median execution time can be found with a low false positive rate given a large sample size of 20 instances.

show abstract

Section: Related Workmentioning

confidence: 99%

Performance testing in the cloud. How bad is it really?

Laaber

Scheuner

Leitner

2018

Preprint

View full text Add to dashboard Cite

show abstract

“…A recent survey by Jiang and Hassan [8] provides a good overview on how research addresses the issue of performance testing of large-scale systems. The authors separate the testing process into three successive steps: test design, execution and analysis and categorize publications along several axes inside each step.…”

Section: Related Workmentioning

confidence: 99%

Performance regression testing and run-time verification of components in robotics systems

Wienke

Wrede

2017

Advanced Robotics

View full text Add to dashboard Cite

Unintended changes in the utilization of resources like CPU and memory can lead to severe problems for the operation of robotics and intelligent systems. Still, systematic testing for such performance regressions has largely been ignored in this domain. We present a method to specify and execute performance tests for individual components of component-based robotics systems based on their component interfaces. The method includes an automatic analysis of each component revision against previous ones that reports potential changes to the resource usage characteristics. This informs developers about the impact of their changes. We describe the design of the framework and present evaluation results for the automatic detection of performance changes based on tests for a variety of robotics components. Additionally, we demonstrate how performance tests can be used as a basis for learning resource utilization models of components. These models can be used to detect faults at system runtime, which provides an additional level of safety for the systems besides the offline testing.

show abstract

“…Secondly, performance test results often depend strongly on the used benchmarking workload, such as the load patterns used for testing. Hence, writing expressive performance tests requires careful identification and modeling of representative workloads or production usage patterns [3,14]. Unfortunately, representative workloads tend to not be as stable as functional unit interfaces.…”

Section: Background and Related Workmentioning

confidence: 99%

An exploratory study of the state of practice of performance testing in Java-based open source projects

Leitner

Bezemer

2016

Preprint

View full text Add to dashboard Cite

An exploratory study of the state of practice of performance testing in Java-based open source projectsThe usage of open source (OS) software is nowadays wide-spread across many industries and domains. While the functional quality of OS projects is considered to be up to par with that of closed-source software, much is unknown about the quality in terms of nonfunctional attributes, such as performance. One challenge for OS developers is that, unlike for functional testing, there is a lack of accepted best practices for performance testing. To reveal the state of practice of performance testing in OS projects, we conduct an exploratory study on 111 Java-based OS projects from GitHub. We study the performance tests of these projects from five perspectives: (1) the developers, (2) size, (3) organization and (4) types of performance tests and (5) the tooling used for performance testing. First, in a quantitative study we show that writing performance tests is not a popular task in OS projects: performance tests form only a small portion of the test suite, are rarely updated, and are usually maintained by a small group of core project developers. Second, we show through a qualitative study that even though many projects are aware that they need performance tests, developers appear to struggle implementing them. We argue that future performance testing frameworks should provider better support for low-friction testing, for instance via non-parameterized methods or performance test generation, as well as focus on a tight integration with standard continuous integration tooling. ABSTRACTThe usage of open source (OS) software is nowadays widespread across many industries and domains. While the functional quality of OS projects is considered to be up to par with that of closed-source software, much is unknown about the quality in terms of non-functional attributes, such as performance. One challenge for OS developers is that, unlike for functional testing, there is a lack of accepted best practices for performance testing.To reveal the state of practice of performance testing in OS projects, we conduct an exploratory study on 111 Java-based OS projects from GitHub. We study the performance tests of these projects from five perspectives: (1) the developers, (2) size, (3) organization and (4) types of performance tests and (5) the tooling used for performance testing.First, in a quantitative study we show that writing performance tests is not a popular task in OS projects: performance tests form only a small portion of the test suite, are rarely updated, and are usually maintained by a small group of core project developers. Second, we show through a qualitative study that even though many projects are aware that they need performance tests, developers appear to struggle implementing them. We argue that future performance testing frameworks should provider better support for lowfriction testing, for instance via non-parameterized methods or performance test generation, as well as focus on a tight integration with standard ...

show abstract

A Survey on Load Testing of Large-Scale Software Systems

Cited by 136 publications

References 138 publications

Performance testing in the cloud. How bad is it really?

Performance testing in the cloud. How bad is it really?

Performance regression testing and run-time verification of components in robotics systems

An exploratory study of the state of practice of performance testing in Java-based open source projects

Contact Info

Product

Resources

About