PHOEBE: an automation framework for the effective usage of diagnosis tools in the performance testing of clustered systems

Abstract

The identification of performance issues and the diagnosis of their root causes are time-consuming and complex tasks, especially in clustered environments. To simplify these tasks, researchers have been developing tools with built-in expertise for practitioners. However, various limitations exist in these tools that prevent their efficient usage in the performance testing of clusters (e.g. the need of manually analysing huge volumes of distributed results). In a previous work, we introduced a policy-based adaptive framework (PHOEBE) that automates the usage of diagnosis tools in the performance testing of clustered systems, in order to improve a tester's productivity, by decreasing the effort and expertise needed to effectively use such tools. This paper extends that work by broadening the set of policies available in PHOEBE, as well as by performing a comprehensive assessment of PHOEBE in terms of its benefits, costs and generality (with respect to the used diagnosis tool). The performed evaluation involved a set of experiments in assessing the different trade-offs commonly experienced by a tester when using a performance diagnosis tool, as well as the time savings that PHOEBE can bring to the performance testing and analysis processes. Our results have shown that PHOEBE can drastically reduce the effort required by a tester to do performance testing and analysis in a cluster. PHOEBE also exhibited consistent behaviour (i.e. similar time-savings and resource utilisations), when applied to a set of commonly used diagnosis tools, demonstrating its generality. Finally, PHOEBE proved to be capable of simplifying the configuration of a diagnosis tool. This was achieved by addressing the identified trade-offs without the need for manual intervention from the tester. PHOEBE is implemented with the multi-agent architecture depicted in Figure 1. There, it can be seen how PHOEBE is composed of three types of agents: the control agent is responsible for interacting with the load testing tool to know when the test starts and ends. It is also responsible for evaluating the policies and propagating the decisions to the other nodes. Meanwhile, the application node agent is responsible for performing the required tasks in each application node (e.g. sampling collection or sending the collected samples to the diagnosis tool). Finally, the diagnosis tool Here, the objective was to evaluate the potential trade-off between the number of samples concurrently processed by a diagnosis tool and the number of resources it requires to process the samples. The following sections describe this experiment and its results. PHOEBE 1863 7.1. Experiment #4: proposed policies assessmentThe objective of this experiment was to evaluate the behaviour of PHOEBE, as well as the set of proposed policies, in order to assess how well they have fulfilled their purpose of addressing the identified trade-offs without the need for manual intervention from the tester. The following sections describe this experiment and its results.7.1.1. Experimental set-up....

One Size Does Not Fit All

“…Performance testing is an important type of testing which aims to assess whether or not an application (i.e., AUT) will be able to perform its business functionality under a given workload [13,19]. As shown in Fig.…”

“…(3) Error rate threshold: This optional parameter defines the upper bound value that indicates that the system has become saturated. If this parameter is not configured, a default value of 90% will be used (as this is a value commonly identified as saturation point [19]). (4) Functional dependencies: This is the set of dependency relationships that might exist between the tested transactions (e.g., a purchase cannot occur without first logging into the system).…”

“…(1) PetStore, an e-commerce application commonly used in the literature [19,22]. It is composed of 11 different business operations.…”

“…In order to do this, WAIT was fed with Javacores [5] (snapshots of the JVM state). They were generated with the native Linux kill command (i.e., no instrumentation was required to create them) and collected every 30 secs (following a sampling interval commonly used in the industry [19]). Finally, a 2-hour test duration (per test run) was used to reflect realistic test conditions.…”

Analysis of Machine Learning Techniques for Anomaly Detection in the Internet of Things

“…Each of the neural network algorithms (LSTM, GRU and SimpleRNN) are trained for 100 epochs before making predictions, and all use the same seeded random values throughout the tests. The data is normalized to small values (between 0 and 10) and an error threshold of 1 (i.e., 10% of the range) is used to predict anomalies (as this threshold is typically used to delimit the behaviour of a steadystate process [15]). For instance, if the predicted value is 0.5 and the actual value is -0.3, it is not considered an anomaly.…”