One Size Does Not Fit All

Abstract

Publisher ACM Link to online version https://icpe2018.spec.org/ Item record/more information http://hdl.handle.net/10197/9960Publisher's statement ABSTRACT Carrying out proper performance testing is considerably challenging. In particular, the identification of performance issues, as well as their root causes, is a time-consuming and complex process which typically requires several iterations of tests (as this type of issues can depend on the input workloads), and heavily relies on human expert knowledge. To improve this process, this paper presents an automated approach (that extends some of our previous work) to dynamically adapt the workload (used by a performance testing tool) during the test runs. As a result, the performance issues of the tested application can be revealed more quickly; hence, identifying them with less effort and expertise. Our experimental evaluation has assessed the accuracy of the proposed approach and the time savings that it brings to testers. The results have demonstrated the benefits of the approach by achieving a significant decrease in the time invested in performance testing (without compromising the accuracy of the test results), while introducing a low overhead in the testing environment.

DYNAMOJM: A JMeter Tool for Performance Testing Using Dynamic Workload Adaptation

“…However, DYNAMO itself requires several configuration parameters which have been implemented in this tool. The accuracy of DYNAMO to find bugs or potential issues has been proved using a proof of concept demo [7]. However, with the DYNAMOJM plugin presented in this paper, it is possible to use the same logic and automation with a friendly graphical interface easy to adopt by performance engineering practitioners.…”

“…If this does not occur, there is a risk that the application under test (AUT) does not exhibit some performance issues and bugs are not found. DYNAMO [7] is an approach that can automatically find an adequate workload for testing without a try and error process, hence, saving time without compromising bug detection. To generate the workload, DYNAMO relies on the analysis of performance samples to dynamically adjust the workload to the maximum allowed without reaching a saturation point in the system.…”

Towards an Efficient Performance Testing Through Dynamic Workload Adaptation

“…Meanwhile, the work presented in [33] describes the performance analysis of software systems as a comparison of two versions of software and their performance results to find possible (regression) bugs. In response to these limitations, approaches that dynamically adapt the workload have been proposed [3,22]. This paper conducts an evaluation of DYNAMO because of its advantages of adjusting the workload on the fly using the analysis of key performance metrics to create a customized workload for the AUT with minimal knowledge or experience from the tester.…”

“…This paper conducts an evaluation of DYNAMO because of its advantages of adjusting the workload on the fly using the analysis of key performance metrics to create a customized workload for the AUT with minimal knowledge or experience from the tester. Moreover, in relation to bug accuracy, previous work [3,22] has proved the benefits of using DYNAMO to find bugs in comparison with static approaches.…”

“…These types of approaches aim to reduce the effort and expertise needed to carry out performance testing efficiently. A representative example of this type of approach is DYNAMO [3], which can automatically find an adequate workload for a web application without requiring multiple test runs while increasing the number of performance bugs found. This is achieved by dynamically defining the appropriate workloads on the fly based on the analysis of the intermediate testing results.…”