The Future of Software Performance Engineering

Woodside, Murray; Franks, Greg; Petriu, Dorina C.

doi:10.1109/fose.2007.32

Cited by 266 publications

(172 citation statements)

References 65 publications

Supporting

Mentioning

161

Contrasting

Unclassified

Order By: Relevance

“…Two general approaches to software performance engineering (SPE) are prevalent: measurement-based SPE, which executes performance experiments and monitors and evaluates their results, and modelbased SPE, which predicts performance characteristics based on the created models [26]. In this paper, we focus on measurement-based SPE.…”

Section: Related Workmentioning

confidence: 99%

Performance testing in the cloud. How bad is it really?

Laaber

Scheuner

Leitner

2018

Preprint

View full text Add to dashboard Cite

Rigorous performance engineering traditionally assumes measuring on bare-metal environments to control for as many confounding factors as possible. Unfortunately, some researchers and practitioners might not have access, knowledge, or funds to operate dedicated performance testing hardware, making public clouds an attractive alternative. However, cloud environments are inherently unpredictable and variable with respect to their performance. In this study, we explore the effects of cloud environments on the variability of performance testing outcomes, and to what extent regressions can still be reliably detected. We focus on software microbenchmarks as an example of performance tests, and execute extensive experiments on three different cloud services (AWS, GCE, and Azure) and for different types of instances. We also compare the results to a hosted bare-metal offering from IBM Bluemix. In total, we gathered more than 5 million unique microbenchmarking data points from benchmarks written in Java and Go. We find that the variability of results differs substantially between benchmarks and instance types (from 0.03% to > 100% relative standard deviation). We also observe that testing using Wilcoxon rank-sum generally leads to unsatisfying results for detecting regressions due to a very high number of false positives in all tested configurations. However, simply testing for a difference in medians can be employed with good success to detect even small differences. In some cases, a difference as low as a 1% shift in median execution time can be found with a low false positive rate given a large sample size of 20 instances.

show abstract

Section: Related Workmentioning

confidence: 99%

Performance testing in the cloud. How bad is it really?

Laaber

Scheuner

Leitner

2018

Preprint

View full text Add to dashboard Cite

show abstract

“…Surveys on model-based performance prediction [1], performance evaluation of component-based systems [12] and the future of software performance engineering [29] provide a broad overview of recent approaches for performance modeling. In addition, Smith and Williams [25] described fundamentals of software performance engineering and Menasce et al [16] detailed on capacity planning with queuing models.…”

Section: Related Workmentioning

confidence: 99%

“…To enable early performance modeling, numerous mathematical notations (e.g., queuing networks, stochastic Petri nets, stochastic process algebra) and software-related notations (e.g., annotated UML) have been proposed [1,29,12]. These notations usually require expertise from the performance domain and their graphical representations are often difficult to discuss with stakeholders from the domain of industrial systems.…”

Section: Introductionmentioning

confidence: 99%

Rapid performance modeling by transforming use case maps to palladio component models

Vogel

Koziolek²,

Goldschmidt³

et al. 2013

Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering

View full text Add to dashboard Cite

Complex information flows in the domain of industrial software systems complicate the creation of performance models to validate the challenging performance requirements. Performance models using annotated UML diagrams or mathematical notations are difficult to discuss with stakeholders from the industrial automation domain, who often have a limited software engineering background. We introduce a novel model transformation from Use Case Maps (UCM) to the Palladio Component Model (PCM), which enables performance modeling based on an intuitive notation for complex information flows. The resulting models can be solved using existing simulators or analytical solvers. We validated the correctness of the transformation with three case study models, and performed a user study. The results showed a performance prediction deviation of less than 10 percent compared to a reference model in most cases.

show abstract

“…Although sound performance analysis theories and techniques exist, they are not widely used because they are difficult to understand and require heavy modeling effort throughout the development process [41]. Consequently, software engineers usually resort to testing to determine whether the performance requirements have been satisfied.…”

Section: Introductionmentioning

confidence: 99%

Model Interoperability for Performance Engineering: Survey of Milestones and Evolution

Smith¹,

Lladó

2011

Performance Evaluation of Computer and Communication Systems. Milestones and Future Challenges

View full text Add to dashboard Cite

Abstract. Next generation Software Performance Engineering tools will exploit a model interoperability paradigm that uses the performance modeling tool best suited to the software/hardware architecture issues and the life cycle stage of the assessment. The paradigm allows the use of existing tools to the extent possible without requiring extensive changes to them. The performance model solution should be transparent to the user. Significant milestones have been accomplished in the evolution of this paradigm. This paper covers key results in the areas of Model Interchange Formats, model transformations, tools, specification of experiments and results, and extensions for real-time and component-based systems. It then offers conclusions on next steps and the future of the model interoperability paradigm.

show abstract

The Future of Software Performance Engineering

Abstract: Abstract

Cited by 266 publications

References 65 publications

Performance testing in the cloud. How bad is it really?

Performance testing in the cloud. How bad is it really?

Rapid performance modeling by transforming use case maps to palladio component models

Model Interoperability for Performance Engineering: Survey of Milestones and Evolution

Contact Info

Product

Resources

About