Performance regression testing target prioritization via performance risk analysis

Huang, Peng; Ma, Xiao; Shen, Dongcai; Zhou, Yuanyuan

doi:10.1145/2568225.2568232

Cited by 66 publications

(61 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A system following a similar basic operating principle is PerfImpact [18], which aims to find changes and input combinations that lead to performance regressions. PRA (performance risk analysis) is an approach used to narow down commits that led to a (previously detected) performance regression [13]. Baltes et al [4] study how developers locate performance problems, and conclude that standard tools do not provide sufficient support for understanding the runtime behavior of software.…”

Section: Background and Related Workmentioning

confidence: 99%

An exploratory study of the state of practice of performance testing in Java-based open source projects

Leitner

Bezemer

2016

Preprint

View full text Add to dashboard Cite

An exploratory study of the state of practice of performance testing in Java-based open source projectsThe usage of open source (OS) software is nowadays wide-spread across many industries and domains. While the functional quality of OS projects is considered to be up to par with that of closed-source software, much is unknown about the quality in terms of nonfunctional attributes, such as performance. One challenge for OS developers is that, unlike for functional testing, there is a lack of accepted best practices for performance testing. To reveal the state of practice of performance testing in OS projects, we conduct an exploratory study on 111 Java-based OS projects from GitHub. We study the performance tests of these projects from five perspectives: (1) the developers, (2) size, (3) organization and (4) types of performance tests and (5) the tooling used for performance testing. First, in a quantitative study we show that writing performance tests is not a popular task in OS projects: performance tests form only a small portion of the test suite, are rarely updated, and are usually maintained by a small group of core project developers. Second, we show through a qualitative study that even though many projects are aware that they need performance tests, developers appear to struggle implementing them. We argue that future performance testing frameworks should provider better support for low-friction testing, for instance via non-parameterized methods or performance test generation, as well as focus on a tight integration with standard continuous integration tooling. ABSTRACTThe usage of open source (OS) software is nowadays widespread across many industries and domains. While the functional quality of OS projects is considered to be up to par with that of closed-source software, much is unknown about the quality in terms of non-functional attributes, such as performance. One challenge for OS developers is that, unlike for functional testing, there is a lack of accepted best practices for performance testing.To reveal the state of practice of performance testing in OS projects, we conduct an exploratory study on 111 Java-based OS projects from GitHub. We study the performance tests of these projects from five perspectives: (1) the developers, (2) size, (3) organization and (4) types of performance tests and (5) the tooling used for performance testing.First, in a quantitative study we show that writing performance tests is not a popular task in OS projects: performance tests form only a small portion of the test suite, are rarely updated, and are usually maintained by a small group of core project developers. Second, we show through a qualitative study that even though many projects are aware that they need performance tests, developers appear to struggle implementing them. We argue that future performance testing frameworks should provider better support for lowfriction testing, for instance via non-parameterized methods or performance test generation, as well as focus on a tight integration with standard ...

show abstract

Section: Background and Related Workmentioning

confidence: 99%

An exploratory study of the state of practice of performance testing in Java-based open source projects

Leitner

Bezemer

2016

Preprint

View full text Add to dashboard Cite

show abstract

“…The TRTs of various transactions are measured and compared with targeted values, typically specified in the service level agreement (SLA). Additionally, it is important to ensure that transactions from the previous version have not regressed .…”

Section: Performance Regression Testing Processmentioning

confidence: 99%

“…The TRTs are measured and compared with targeted values as per the SLA). Performance regression testing run (PRTR): The TRTs of the PPTR cannot be compared with TRTs from previous release because of different workloads (number of users and the transaction mix) . Hence, the PRTR is performed on the new release but with a workload similar to that used in the previous release . This run is compared with previous release run as depicted in Figure .…”

Section: Performance Regression Testing Processmentioning

confidence: 99%

Anomaly detection in performance regression testing by transaction profile estimation

Ghaith

Wang

Perry

et al. 2015

Software Testing Verif & Rel

View full text Add to dashboard Cite

Summary As part of the process to test a new release of an application, the performance testing team need to confirm that the existing functionalities do not perform worse than those in the previous release, a problem known as performance regression anomaly. Most existing approaches to analyse performance regression testing data vary according to the applied workload, which usually leads to the need for an extra performance testing run. To ease such lengthy tasks, we propose a new workload‐independent, automated technique to detect anomalies in performance regression testing data using the concept known as transaction profile (TP). The TP is inferred from the performance regression testing data along with the queueing network model of the testing system. Based on a case study conducted against two web applications, one open source and one industrial, we have been able to automatically generate the ‘TP run report’ and verify that it can be used to uncover performance regression anomalies caused by software updates. In particular, the report helped us to isolate the real anomaly issues from those caused by workload changes with an average F1 measure of 85% for the open source application and 90% for the industrial application. Such results support our proposal to use the TP as a more efficient technique in identifying performance regression anomalies than the state of the art industry and research techniques. Copyright © 2015 John Wiley & Sons, Ltd.

show abstract

“…Search‐based software engineering (SBSE) is a technique for solving software engineering problems using search algorithms. In SBSE, many algorithms have been proposed, with focus on multiobjective TCP (MoTCP) more recently, including evolutionary computation (EC)–based algorithms such as non‐dominated sorted genetic algorithm‐II (NSGA‐II), the two‐archive evolution algorithm (TAEA), and the strength Pareto evolution algorithm 2 (SPEA2) . Swarm intelligent (SI) algorithms such as particle swarm optimization (PSO) have also been used in MoTCP.…”

Section: Introductionmentioning

confidence: 99%

“…Test case prioritization is proposed to reorder the test cases execution sequence to discover errors early. Many search‐based optimization techniques for TCP have been presented …”

Section: Introductionmentioning

confidence: 99%

Concrete hyperheuristic framework for test case prioritization

Bian

Zheng

Guo

et al. 2018

J Software Evolu Process

View full text Add to dashboard Cite

Test case prioritization (TCP), which aims to find the optimal test case execution sequences for specific testing objects, has been widely used in regression testing. A wide variety of search methodologies and algorithms have been proposed to optimize test case execution sequences, namely, search‐based TCP. However, different algorithms perform differently and have different implementation costs and specific situations where an algorithm usually performs with high effectiveness and efficiency. When facing a new testing scenario, it is actually difficult to decide which algorithm is suitable. In this paper, to address the algorithm selection problem for different test scenarios, a more generally applicable algorithm based on a hyperheuristic strategy is proposed for search‐based TCP. This includes a range of multiobjective algorithms with a variety of crossover strategies and a learning agent strategy to evaluate and select the appropriate algorithm execution sequence dynamically for different scenarios. The concrete hyperheuristic framework for multiobjective TCP is presented with an algorithm's repository in the low level and the learning agent strategy in the higher level. Experiments show that the proposed learning agent strategy can accurately evaluate algorithms in multiobjective problems and select the appropriate algorithm in each iteration.

show abstract

Performance regression testing target prioritization via performance risk analysis

Cited by 66 publications

References 27 publications

An exploratory study of the state of practice of performance testing in Java-based open source projects

An exploratory study of the state of practice of performance testing in Java-based open source projects

Anomaly detection in performance regression testing by transaction profile estimation

Concrete hyperheuristic framework for test case prioritization

Contact Info

Product

Resources

About