Continuous validation of performance test workloads

Syer, Mark D.; Shang, Weiyi; Jiang, Zhen Ming; Hassan, Ahmed E.

doi:10.1007/s10515-016-0196-8

Cited by 19 publications

(18 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Prior research proposes approaches for recovering workloads to assist in the design of load tests Vögele, van Hoorn, Schulz, Hasselbring, and Krcmar (2018), validating whether load tests are field representative as production Syer et al (2017), optimizing system performance (Summers, Brecht, Eager, & Gutarin, 2016;H. Xi et al, 2011) and detecting system performance issues (Cohen et al, 2005;Hassan, Martin, Flora, Mansfield, & Dietz, 2008;Shang et al, 2015;Syer et al, 2017). All the above prior work illustrates the value and importance of recovering representative workloads.…”

Section: Recovering Workloadmentioning

confidence: 99%

“…Hadoop performs data processing in a reliable, efficient, high fault tolerance, low cost, and scalable manner. We choose Hadoop since it is highly concerned with its performance and has been studied in prior research in mining performance data Syer et al (2017). RxJava is a library for composing asynchronous and event-based programs by using observable sequences and it carries the JMH benchmarks test options.…”

Section: Subject Systemsmentioning

confidence: 99%

“…We use each metric as a dependent variable and use the rest of the metrics as independent variables to build a regression model. We calculate the R 2 of each model and if the R 2 is larger than a threshold (0.9) Syer et al (2017), the current dependent variable is considered redundant. We then remove the metric with the highest R 2 and repeat the process until no metric can be predicted with R 2 higher than the threshold.…”

Section: Data Preprocessingmentioning

confidence: 99%

“…OpenJPA is an open-source system from the Apache organization that implements the JPA standard. We choose the three subject systems since they are highly concerned with their performance and have been studied in prior research in mining performance data (T.-H. Chen et al, 2014;Syer et al, 2017). The overview of the three subject systems is shown in Table 4.2.…”

Section: Subject Systemsmentioning

confidence: 99%

“…Hadoop White ( 2009) is a distributed system infrastructure, whereas Cassandra is a distributed NoSQL database management system. We choose these two subject systems since their performance is critical for the users and they have been studied in prior research on mining performance data (T.-H. Chen et al, 2014;Syer et al, 2017). In addition, they have a large number of configuration options.…”

Section: Subject Systemsmentioning

confidence: 99%

See 4 more Smart Citations

Performance regression detection in DevOps

Chen

2020

Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Companion Proceedings

View full text Add to dashboard Cite

Performance is an important aspect of software quality. The goals of performance are typically defined by setting upper and lower bounds for response time and throughput of a system and physical level measurements such as CPU, memory, and I/O. To meet such performance goals, several performance-related activities are needed in development (Dev) and operations (Ops). Large software system failures are often due to performance issues rather than functional bugs. One of the most important performance issues is performance regression. Although performance regressions are not all bugs, they often have a direct impact on users' experience of the system. The process of detection of performance regressions in development and operations is faced with challenges. First, the detection of performance regression is conducted after the fact, i.e., after the system is built and deployed in the field or dedicated performance testing environments. Large amounts of resources are required to detect, locate, understand, and fix performance regressions at such a late stage in the development cycle. Second, even we can detect a performance regression, it is extremely hard to fix it because other changes are applied to the system after the introduction of the regression.These challenges call for further in-depth analyses of the performance regression. In this thesis, to avoid performance regression slipping into operation, we first perform an exploratory study on the source code changes that introduce performance regressions in order to understand root-causes of performance regression in the source code level. Second, we propose an approach that automatically predicts whether a test would manifest performance regressions in a code commit.Most of the performance issues are related to configurations. Therefore, third, we propose an iii approach that predicts whether a configuration option manifests a performance variation issue. To assist practitioners to analyze system performance with operational data, we propose an approach to recovering field-representative workload that can be used to detect performance regression.ivContents List of Figures xvi List of Tables xviii mance counters. Improvement are calculated by comparing with a random classifier.

show abstract

Section: Recovering Workloadmentioning

confidence: 99%

Section: Subject Systemsmentioning

confidence: 99%

Section: Data Preprocessingmentioning

confidence: 99%

Section: Subject Systemsmentioning

confidence: 99%

Section: Subject Systemsmentioning

confidence: 99%

See 3 more Smart Citations

Performance regression detection in DevOps

Chen

2020

Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Companion Proceedings

View full text Add to dashboard Cite

show abstract

On the Performance Implications of Deploying IoT Apps as FaaS

Aly

Khomh

Yacout

2022

Communications in Computer and Information Science

View full text Add to dashboard Cite

Empirical study on the discrepancy between performance testing results from virtual and physical environments

2017

Self Cite

View full text Add to dashboard Cite

An Empirical Study on the Discrepancy between Performance Testing Results from Virtual and Physical EnvironmentsMuhammad Moiz Arif Large software systems often undergo performance tests to ensure their capability to handle expected loads. These performance tests often consume large amounts of computing resources and time in order to exercise the system extensively and build confidence on results. Making it worse, the ever evolving field environments require frequent updates to the performance testing environment. In practice, virtual machines (VMs) are widely exploited to provide flexible and less costly environments for performance tests. However, the use of VMs may introduce confounding overhead (e.g., a higher than expected memory utilization with unstable I/O traffic) to the testing environment and lead to unrealistic performance testing results. Yet, little research has studied the impact on test results of using VMs in performance testing activities.In this thesis, we evaluate the discrepancy between the performance testing results from virtual and physical environments. We perform a case study on two open source systems -namely Dell DVD Store (DS2) and CloudStore. We conduct the same performance tests in both virtual and physical environments and compare the performance testing results based on the three aspects that are typically examined for performance testing results: 1) single performance metric (e.g. CPU usage from virtual environment vs. CPU usage from physical environment), 2) the relationship between two performance metrics (e.g. correlation between CPU usage and I/O traffic) and iii Contents List of Figures xi List of Tables xiiiList of Tables Spearman's rank correlation coefficients and p-values of the highlighted

show abstract

Continuous validation of performance test workloads

Cited by 19 publications

References 42 publications

Performance regression detection in DevOps

Performance regression detection in DevOps

On the Performance Implications of Deploying IoT Apps as FaaS

Empirical study on the discrepancy between performance testing results from virtual and physical environments

Contact Info

Product

Resources

About