An Experimental Performance Evaluation of Autoscalers for Complex Workflows

Abstract

Elasticity is one of the main features of cloud computing allowing customers to scale their resources based on the workload. Many autoscalers have been proposed in the past decade to decide on behalf of cloud customers when and how to provision resources to a cloud application based on the workload utilizing cloud elasticity features. However, in prior work, when a new policy is proposed, it is seldom compared to the state-of-the-art, and is often compared only to static provisioning using a predefined QoS target. This reduces the ability of cloud customers and of cloud operators to choose and deploy an autoscaling policy as there is seldom enough analysis on the performance of the autoscalers in different operating conditions and with different applications. In our work, we conduct an experimental performance evaluation of autoscaling policies, using as workflows, a commonly used formalism for automating resource management for applications with well-defined yet complex structures. We present a detailed comparative study of general state-of-the-art autoscaling policies, along with two new workflow-specific policies. To understand the performance differences between the seven policies, we conduct various experiments and compare their performance in both pairwise and group comparisons. We report both individual and aggregated metrics. As many workflows have deadline requirements on the tasks, we study the effect of autoscaling on workflow deadlines. Additionally, we look into the effect of autoscaling on the accounted and hourly-based charged costs, and evaluate performance variability caused by the autoscaler selection for each group of workflow sizes. Our results highlight the trade-offs between the suggested policies, how they can impact meeting the deadlines, and how they perform in different operating conditions, thus enabling a better understanding of the current state-of-the-art. CCS Concepts: •General and reference →Cross-computing tools and techniques; •Networks →Cloud computing; •Computer systems organization →Self-organizing autonomic computing; •Software and its engineering →Virtual machines; ACM Transactions on Modeling and Performance Evaluation of Computing Systems This is a significantly extended and more comprehensive version of our ICPE 2017 article [ 28].

Chamulteon: Coordinated Auto-Scaling of Micro-Services

“…This effect can be seen until 900 seconds as the resource supply for each service is increasing slower than the previous service supplies. Similar to the observations in the articles [2], [27], [30], Reg exhibits a high rate of oscillations (between second 1000 and 1600) that cannot be explained. After second 2000, Reg tends to over-provision without any oscillations.…”

“…Although our experimental analysis covers different scenarios and we compare different auto-scalers, the results may not be generalizable to other types of applications or to closedsource auto-scalers. In principle, for the evaluated competing auto-scalers, a comparable behavior has been observed in the related work on auto-scaler evaluation [2], [27], [30].…”

An Experimental Evaluation of the Kubernetes Cluster Autoscaler in the Cloud

“…To enable comparison of novel autoscaling methods not only to static provisioning as done in the past, but also to other autoscaling algorithms, the SPEC Cloud Group developed a set of standard autoscaling performance metrics [7] that are now being used by a number of works for comparing multiple autoscalers. Ilyushkin et al [14] use these metrics to compare seven autoscaling policies from the state of the art, whereas Versluis et al [15] present a simulation-based experimental evaluation of autoscaling workloads of workflows in data centers. These works provide a better understanding of the performance of autoscaling policies proposed in the past decade.…”

“…Our work extends these works by using similar metrics to quantify the performance of the autoscaling policy of the Kubernetes platform which is popular in the cloud-native application development paradigm. However, [14] focuses on scientific workflows while [15] focuses on scientific, industrial and engineering workflows. In contrast, we focus on two representative containerized applications for Kubernetes.…”

Self-Adaptive Data Processing to Improve SLOs for Dynamic IoT Workloads

“…The reason behind having separate calculation schemes is due to the necessity of having instances ready just in time for the coming workload when conducting the scale-out; thus, the calculation is better done in advance. On the other hand, the scale-in (or reduction in compute resources) is appreciated only in case the deployed application and the virtual infrastructure would not have to scale-out again within a meaningful amount of time; this amount of time has to outweigh the cost of scaling in and out again [19]. The equation that is used to calculate the desired amount of data processor instances for the IoT platform which capitalizes on the idea of computing the scaled-out and scaled-in amount of instances differently is shown below (see Equation (2)).…”