Finding the right cloud configuration for analytics clusters

Abstract

Finding good cloud configurations for deploying a single distributed system is already a challenging task, and it becomes substantially harder when a data analytics cluster is formed by multiple distributed systems since the search space becomes exponentially larger. In particular, recent proposals for single system deployments rely on benchmarking runs that become prohibitively expensive as we shift to joint optimization of multiple systems, as users have to wait until the end of a long optimization run to start the production run of their job. We propose Vanir, an optimization framework designed to operate in an ecosystem of multiple distributed systems forming an analytics cluster. To deal with this large search space, Vanir takes the approach of quickly finding a good enough configuration and then attempts to further optimize the configuration during production runs. This is achieved by combining a series of techniques in a novel way, namely a metrics-based optimizer for the benchmarking runs, and a Mondrian forest-based performance model and transfer learning during production runs. Our results show that Vanir can find deployments that perform comparably to the ones found by state-of-the-art single-system cloud configuration optimizers while spending 2× fewer benchmarking runs. This leads to an overall search cost that is 1.3-24× lower compared to the state-of-the-art. Additionally, when transfer learning can be used, Vanir can minimize the benchmarking runs even further, and use online optimization to achieve a performance comparable to the deployments found by today's single-system frameworks. * Work done in part while author was interning at KAUST.

Best VM Selection for Big Data Applications across Multiple Frameworks by Transfer Learning

“…We use default values to set application parameters except for executor 4 and memory parameters in Spark. In order to prevent out of memory (OOM) exceptions, we use Mesos [13] to watch the real usage of memory per executor.…”

“…However, we have illustrated that the way of reusing pre-trained model is fragile when workloads are from different frameworks. Vanir [4] combines a series of techniques such as Mondrian forest model and transfer learning to search the right cloud configurations. But their have not yet studied the correlation similarities or other types of similarities across frameworks.…”

“…To address this challenge, existing performance modeling efforts [21,25,29] and machine learning approaches [4,18,28] have to tolerate huge offline training overhead to build an accurate online model for each framework, since they just consider low-level metrics (such as resource utilizations) within a framework. Sadly, they have to spend a lot of time to train new models for similar applications for new frameworks, although recent works [3,5,10] have proved that these similar applications, both in Hadoop and Spark, involve a wide range of use cases (micro benchmark, machine learning, stream processing and etc.).…”

“…Here, we can see their heat maps of budget look completely different. This implies that existing approaches [4,21,25,28,29] may have to build one model per framework. Fortunately, we observed that low-level metrics have high-level similarities across frameworks (blue area), the best (or near best) VM types all appear in the area which follows similar CPU-to-memory ratio (e.g., 8G8U, 16G16U).…”

Accelerating the Configuration Tuning of Big Data Analytics with Similarity-aware Multitask Bayesian Optimization

“…Some configuration tuning systems have leveraged similarity to accelerate the tuning process: AROMA [11] clusters the Hadoop workloads then builds a performance model that guides the tuning of each workload cluster. Scout [43] and Vanir [44] exploit workload similarity to explore the search space more effectively for tuning cloud configurations (number of instances and their resource allocation). Ultimately, a joint solution for optimizing cloud instance and DISC framework configurations will be needed.…”