Experience with performance testing of software systems: issues, an approach, and case study

Weyuker, Elaine J.; Vokolos, Filippos I.

doi:10.1109/32.888628

Cited by 202 publications

(112 citation statements)

References 4 publications

Supporting

Mentioning

106

Contrasting

Unclassified

Order By: Relevance

“…First and foremost, a recurring observation in our study is that performance testing in Java-based OS software is neither as extensive nor as standardized as it is the case for functional testing. While the lack of standardization was already observed by Weyuker et al in 2000 [25], our study results indicate that even at the time of writing there is still a long way to go before performance engineering truly becomes an integral part of the software development lifecycle for OS software. We speculate that one reason for this evident lack of standardization is that there currently is no "killer app" to conduct performance tests with in an easy, yet powerful, way, similar to how JUnit has standardised unit testing for Java.…”

Section: Discussionmentioning

confidence: 39%

See 1 more Smart Citation

An exploratory study of the state of practice of performance testing in Java-based open source projects

Leitner

Bezemer

2016

Preprint

View full text Add to dashboard Cite

An exploratory study of the state of practice of performance testing in Java-based open source projectsThe usage of open source (OS) software is nowadays wide-spread across many industries and domains. While the functional quality of OS projects is considered to be up to par with that of closed-source software, much is unknown about the quality in terms of nonfunctional attributes, such as performance. One challenge for OS developers is that, unlike for functional testing, there is a lack of accepted best practices for performance testing. To reveal the state of practice of performance testing in OS projects, we conduct an exploratory study on 111 Java-based OS projects from GitHub. We study the performance tests of these projects from five perspectives: (1) the developers, (2) size, (3) organization and (4) types of performance tests and (5) the tooling used for performance testing. First, in a quantitative study we show that writing performance tests is not a popular task in OS projects: performance tests form only a small portion of the test suite, are rarely updated, and are usually maintained by a small group of core project developers. Second, we show through a qualitative study that even though many projects are aware that they need performance tests, developers appear to struggle implementing them. We argue that future performance testing frameworks should provider better support for low-friction testing, for instance via non-parameterized methods or performance test generation, as well as focus on a tight integration with standard continuous integration tooling. ABSTRACTThe usage of open source (OS) software is nowadays widespread across many industries and domains. While the functional quality of OS projects is considered to be up to par with that of closed-source software, much is unknown about the quality in terms of non-functional attributes, such as performance. One challenge for OS developers is that, unlike for functional testing, there is a lack of accepted best practices for performance testing.To reveal the state of practice of performance testing in OS projects, we conduct an exploratory study on 111 Java-based OS projects from GitHub. We study the performance tests of these projects from five perspectives: (1) the developers, (2) size, (3) organization and (4) types of performance tests and (5) the tooling used for performance testing.First, in a quantitative study we show that writing performance tests is not a popular task in OS projects: performance tests form only a small portion of the test suite, are rarely updated, and are usually maintained by a small group of core project developers. Second, we show through a qualitative study that even though many projects are aware that they need performance tests, developers appear to struggle implementing them. We argue that future performance testing frameworks should provider better support for lowfriction testing, for instance via non-parameterized methods or performance test generation, as well as focus on a tight integration with standard ...

show abstract

Section: Discussionmentioning

confidence: 39%

“…Early on, studies have reported that industrial practice in performance testing is not on the same level as functional testing [25], even for large, performance-sensitive enterprise applications. Historically, performance testing has been made difficult by two peculiarities.…”

Section: Background and Related Workmentioning

confidence: 99%

An exploratory study of the state of practice of performance testing in Java-based open source projects

Leitner

Bezemer

2016

Preprint

View full text Add to dashboard Cite

show abstract

“…Besides validating a particular methodology, the question of prediction precision is tackled from many other angles, including workload characterization [29,16], or generation of either the executable system [31,13,33,6] or the performance model [15,19]. What these approaches have in common is that they still require Self-archived copy.…”

Section: Related Workmentioning

confidence: 99%

Validating Model-Driven Performance Predictions on Random Software Systems

Babka

Tůma

Bulej

2010

Research Into Practice – Reality and Gaps

View full text Add to dashboard Cite

Abstract. Software performance prediction methods are typically validated by taking an appropriate software system, performing both performance predictions and performance measurements for that system, and comparing the results. The validation includes manual actions, which makes it feasible only for a small number of systems. To significantly increase the number of systems on which software performance prediction methods can be validated, and thus improve the validation, we propose an approach where the systems are generated together with their models and the validation runs without manual intervention. The approach is described in detail and initial results demonstrating both its benefits and its issues are presented.Key words: performance modeling, performance validation, MDD MotivationState of the art in model-driven software performance prediction builds on three related factors: the availability of architectural and behavioral software models, the ability to solve performance models, and the ability to transform the former models into the latter. This is illustrated for example by the survey of modeldriven software performance prediction [3], which points out that the typical approach is to use UML diagrams for specifying both the architecture and the behavior of the software system, and to transform these diagrams into performance models based on queueing networks.Both the models and the methods involved in the prediction process necessarily include simplifying assumptions that help abstract away from some of the complexities of the modeled system, e.g., approximating real operation times with probability distributions or assuming independence of operation times. These simplifications are necessary to make the entire prediction process tractable, but the complexity of the modeled system usually makes it impossible to say how the simplifications influence the prediction precision.Self-archived copy. The original publication is available at www.springerlink.com,

show abstract

“…Traditionally, performance testing research was conducted in the context of system-scale load and stress testing [3,14,19,25]. By now, such performance tests are academically well-understood, and recent research focuses on industrial applicability [8,21] or how to reduce the time necessary for load testing [11].…”

Section: Related Workmentioning

confidence: 99%

Performance testing in the cloud. How bad is it really?

Laaber

Scheuner

Leitner

2018

Preprint

View full text Add to dashboard Cite

Rigorous performance engineering traditionally assumes measuring on bare-metal environments to control for as many confounding factors as possible. Unfortunately, some researchers and practitioners might not have access, knowledge, or funds to operate dedicated performance testing hardware, making public clouds an attractive alternative. However, cloud environments are inherently unpredictable and variable with respect to their performance. In this study, we explore the effects of cloud environments on the variability of performance testing outcomes, and to what extent regressions can still be reliably detected. We focus on software microbenchmarks as an example of performance tests, and execute extensive experiments on three different cloud services (AWS, GCE, and Azure) and for different types of instances. We also compare the results to a hosted bare-metal offering from IBM Bluemix. In total, we gathered more than 5 million unique microbenchmarking data points from benchmarks written in Java and Go. We find that the variability of results differs substantially between benchmarks and instance types (from 0.03% to > 100% relative standard deviation). We also observe that testing using Wilcoxon rank-sum generally leads to unsatisfying results for detecting regressions due to a very high number of false positives in all tested configurations. However, simply testing for a difference in medians can be employed with good success to detect even small differences. In some cases, a difference as low as a 1% shift in median execution time can be found with a low false positive rate given a large sample size of 20 instances.

show abstract

Experience with performance testing of software systems: issues, an approach, and case study

Cited by 202 publications

References 4 publications

An exploratory study of the state of practice of performance testing in Java-based open source projects

An exploratory study of the state of practice of performance testing in Java-based open source projects

Validating Model-Driven Performance Predictions on Random Software Systems

Performance testing in the cloud. How bad is it really?

Contact Info

Product

Resources

About