An Experimental Performance Evaluation of Autoscaling Policies for Complex Workflows

Abstract

Simplifying the task of resource management and scheduling for customers, while still delivering complex Quality-of-Service (QoS), is key to cloud computing. Many autoscaling policies have been proposed in the past decade to decide on behalf of cloud customers when and how to provision resources to a cloud application utilizing cloud elasticity features. However, in prior work, when a new policy is proposed, it is seldom compared to the state-of-the-art, and is often compared only to static provisioning using a predefined QoS target. This reduces the ability of cloud customers and of cloud operators to choose and deploy an autoscaling policy. In our work, we conduct an experimental performance evaluation of autoscaling policies, using as application model workflows, a commonly used formalism for automating resource management for applications with well-defined yet complex structure. We present a detailed comparative study of general state-of-the-art autoscaling policies, along with two new workflow-specific policies. To understand the performance differences between the 7 policies, we conduct various forms of pairwise and group comparisons. We report both individual and aggregated metrics. Our results highlight the trade-offs between the suggested policies, and thus enable a better understanding of the current state-of-the-art.

A Cloud-aware Autonomous Workflow Engine and Its Application to Gene Regulatory Networks Inference

“…A comparative evaluation of several auto-scaling algorithms is given in (Ilyushkin et al, 2017). However, the policies discussed in the review solely focus on auto-scaling, thus missing on issues like data locality and tasks clustering.…”

Ananke: A Q-Learning-Based Portfolio Scheduler for Complex Industrial Workflows

“…We embed some of their policies as part of the portfolio used by ANANKE for auto-scaling, and in general extend their work through the Q-learning and portfolio scheduling structure. Ilyushkin et al [21] propose a de-tailed comparative study of a set of auto-scaling algorithms. We use their systemand user-oriented evaluation metrics to assess the performance of our auto-scaling approach, but consider different workloads and thus supply and demand curves.…”

“…To evaluate elasticity, we adopt the metrics and comparison approaches introduced in 2017 by the SPEC Cloud Group [21].…”

“…Elasticity metrics: We adopt the suite of metrics proposed by the SPEC Cloud Group [21], each of which characterizes a different facet of the mismatch between supply and demand of resources. We use three classes of system-oriented elasticity metrics, related to accuracy, duration of incorrect provisioning, and instability caused by elasticity decisions.…”

Companion Proceedings of the10th International Conference on Utility and Cloud Computing

“…First, the application provider will have to manually tweak several combinations of metrics to determine empirically a suitable configuration of initial scaling factors and metric thresholds. Second, most auto-scaling systems are simplistic in the sense of treating all microservices equally without the consideration of dependencies or specificities including resource quotas [4,10].…”