Machines Tuning Machines: Configuring Distributed Stream Processors with Bayesian Optimization

Abstract

Modern distributed computing frameworks such as Apache Hadoop, Spark, or Storm distribute the workload of applications across a large number of machines. Whilst they abstract the details of distribution they do require the programmer to set a number of configuration parameters before deployment. These parameter settings (usually) have a substantial impact on execution efficiency. Finding the right values for these parameters is considered a difficult task and requires domain, application, and framework expertise. In this paper, we propose a machine learning approach to the problem of configuring a distributed computing framework. Specifically, we propose using Bayesian Optimization to find good parameter settings. In an extensive empirical evaluation, we show that Bayesian Optimization can effectively find good parameter settings for four different stream processing topologies implemented in Apache Storm resulting in significant gains over a parallel linear approach. Abstract-Modern distributed computing frameworks such as Apache Hadoop, Spark, or Storm distribute the workload of applications across a large number of machines. Whilst they abstract the details of distribution they do require the programmer to set a number of configuration parameters before deployment. These parameter settings (usually) have a substantial impact on execution efficiency. Finding the right values for these parameters is considered a difficult task and requires domain, application, and framework expertise.In this paper, we propose a machine learning approach to the problem of configuring a distributed computing framework. Specifically, we propose using Bayesian Optimization to find good parameter settings. In an extensive empirical evaluation, we show that Bayesian Optimization can effectively find good parameter settings for four different stream processing topologies implemented in Apache Storm resulting in significant gains over a parallel linear approach.

Optimizing a Gamified Design Through Reinforcement Learning - a Case Study in Stack Overflow

“…A model-based approach is concerned with conducting experiments on a chosen set of configurations to observe their performance [11,30,26,15,24]. The restriction of using a limited number of configurations is a limitation that our approach overcomes because there are numerous configurations in practice with many dependencies between them and these can be used to get better optimization.…”

A Survey on Automatic Parameter Tuning for Big Data Processing Systems

“…Such settings are expected to improve load balancing in the cluster. Other suggestions include maintaining a ratio of one task per Executor to prevent the context switching overhead among tasks, as well as having one Acker thread per Worker process [41]. Besides, the total number of CPU-bound tasks should not exceed the total number of Workers to avoid CPU contention, while I/O-bound tasks could exceed that limit [1].…”

“…Fischer et al [41] proposed a method to automatically choose configuration for distributed stream processing. In particular, they leverage a Bayesian network to predict the parameters with good performance.…”

Effectively Testing System Configurations of Critical IoT Analytics Pipelines

“…The results are used to train a statistical model for finding good configurations. Fischer et al [14] and Trotter et al [17] present an auto-tuning algorithm using Bayesian Optimization (BO) [19] to achieve high throughput. Jamshidi et al [15] likewise proposes BO, however, it optimizes latency and leverages Gaussian Processes [20] to continuously estimate the mean and confidence interval of a response variable at yet-to-be explored configurations.…”