Towards an automated approach to use expert systems in the performance testing of distributed systems

Publisher ACM Link to online version https://icpe2018.spec.org/ Item record/more information http://hdl.handle.net/10197/9960Publisher's statement ABSTRACT Carrying out proper performance testing is considerably challenging. In particular, the identification of performance issues, as well as their root causes, is a time-consuming and complex process which typically requires several iterations of tests (as this type of issues can depend on the input workloads), and heavily relies on human expert knowledge. To improve this process, this paper presents an automated approach (that extends some of our previous work) to dynamically adapt the workload (used by a performance testing tool) during the test runs. As a result, the performance issues of the tested application can be revealed more quickly; hence, identifying them with less effort and expertise. Our experimental evaluation has assessed the accuracy of the proposed approach and the time savings that it brings to testers. The results have demonstrated the benefits of the approach by achieving a significant decrease in the time invested in performance testing (without compromising the accuracy of the test results), while introducing a low overhead in the testing environment.

show abstract

“…(1) PetStore, an e-commerce application commonly used in the literature [19,22]. It is composed of 11 different business operations.…”

Section: Experimental Evaluation 41 Experimental Setupmentioning

confidence: 99%

One Size Does Not Fit All

Ayala-Rivera

Kaczmarski

Murphy

et al. 2018

Proceedings of the 2018 ACM/SPEC International Conference on Performance Engineering

Self Cite

View full text Add to dashboard Cite

show abstract

“…First, we provide the context of our solution. Next, we describe the internal components of PHOEBE (preliminary versions of these components, applied to a single diagnosis tool, were presented in ).…”

Section: Phoebe: An Automation Framework For Performance Testingmentioning

confidence: 99%

“…The aim has been to improve a tester's productivity by decreasing the effort and expertise needed to use diagnosis tools. In our previous work , we presented PHOEBE, an adaptive framework that automates the configuration and usage of a diagnosis tool in a clustered testing environment (typically located in a data centre). PHOEBE is shown in Figure .…”

Section: Introductionmentioning

confidence: 99%

PHOEBE: an automation framework for the effective usage of diagnosis tools in the performance testing of clustered systems

et al. 2017

Self Cite

View full text Add to dashboard Cite

The identification of performance issues and the diagnosis of their root causes are time-consuming and complex tasks, especially in clustered environments. To simplify these tasks, researchers have been developing tools with built-in expertise for practitioners. However, various limitations exist in these tools that prevent their efficient usage in the performance testing of clusters (e.g. the need of manually analysing huge volumes of distributed results). In a previous work, we introduced a policy-based adaptive framework (PHOEBE) that automates the usage of diagnosis tools in the performance testing of clustered systems, in order to improve a tester's productivity, by decreasing the effort and expertise needed to effectively use such tools. This paper extends that work by broadening the set of policies available in PHOEBE, as well as by performing a comprehensive assessment of PHOEBE in terms of its benefits, costs and generality (with respect to the used diagnosis tool). The performed evaluation involved a set of experiments in assessing the different trade-offs commonly experienced by a tester when using a performance diagnosis tool, as well as the time savings that PHOEBE can bring to the performance testing and analysis processes. Our results have shown that PHOEBE can drastically reduce the effort required by a tester to do performance testing and analysis in a cluster. PHOEBE also exhibited consistent behaviour (i.e. similar time-savings and resource utilisations), when applied to a set of commonly used diagnosis tools, demonstrating its generality. Finally, PHOEBE proved to be capable of simplifying the configuration of a diagnosis tool. This was achieved by addressing the identified trade-offs without the need for manual intervention from the tester. PHOEBE is implemented with the multi-agent architecture depicted in Figure 1. There, it can be seen how PHOEBE is composed of three types of agents: the control agent is responsible for interacting with the load testing tool to know when the test starts and ends. It is also responsible for evaluating the policies and propagating the decisions to the other nodes. Meanwhile, the application node agent is responsible for performing the required tasks in each application node (e.g. sampling collection or sending the collected samples to the diagnosis tool). Finally, the diagnosis tool Here, the objective was to evaluate the potential trade-off between the number of samples concurrently processed by a diagnosis tool and the number of resources it requires to process the samples. The following sections describe this experiment and its results. PHOEBE 1863 7.1. Experiment #4: proposed policies assessmentThe objective of this experiment was to evaluate the behaviour of PHOEBE, as well as the set of proposed policies, in order to assess how well they have fulfilled their purpose of addressing the identified trade-offs without the need for manual intervention from the tester. The following sections describe this experiment and its results.7.1.1. Experimental set-up....

show abstract

“…Finally, other efforts have also centred on reducing the expertise required. For instance, by automating the usage of diagnosis tools [17,18]. Unlike these works, which have been designed to improve other aspects of performance testing, our approach has been designed to address the specific need of a tester to set an appropriate test workload, hence isolating her from the complexities of identifying it.…”

Section: Background and Related Workmentioning

confidence: 99%

“…One VM used an Apache JMeter 2.9 (a leading open-source tool for performance testing), and the other VM ran an Apache Tomcat 6.0.35 (a popular open-source Java Web Application Server) [2]. As the application to test, we used JPetStore [5], an open-source e-commerce application, which is commonly used in the literature [17,18]. Also, we used IBM WAIT as diagnosis tool due to its strong analytic capabilities to detect performance issues (e.g., lock contention or database bottleneck) in Java systems [8].…”

Section: Experimental Evaluationmentioning

confidence: 99%

In-Test Adaptation of Workload in Enterprise Application Performance Testing

Kaczmarski

Perry

Murphy

et al. 2017

Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering Companion

Self Cite

View full text Add to dashboard Cite

Performance testing is used to assess if an enterprise application can fulfil its expected Service Level Agreements. However, since some performance issues depend on the input workloads, it is common to use time-consuming and complex iterative test methods, which heavily rely on human expertise. This paper presents an automated approach to dynamically adapt the workload so that issues (e.g. bottlenecks) can be identified more quickly as well as with less effort and expertise. We present promising results from an initial validation prototype indicating an 18-fold decrease in the test time without compromising the accuracy of the test results, while only introducing a marginal overhead in the system.

show abstract

Towards an automated approach to use expert systems in the performance testing of distributed systems

Cited by 5 publications

References 15 publications

One Size Does Not Fit All

One Size Does Not Fit All

PHOEBE: an automation framework for the effective usage of diagnosis tools in the performance testing of clustered systems

In-Test Adaptation of Workload in Enterprise Application Performance Testing

Contact Info

Product

Resources

About