Bellamy: Reusing Performance Models for Distributed Dataflow Jobs Across Contexts

Scheinert, Dominik; Thamsen, Lauritz; Zhu, Houkun; Will, Jonathan; Acker, Alexander; Wittkopp, Thorsten; Kao, Odej

doi:10.1109/cluster48925.2021.00052

Cited by 17 publications

(19 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They often use runtime data to predict the scale-out and runtime behavior of jobs. The runtime data can be gained either from dedicated profiling with a sample of the dataset or previous full executions [19], [20], [23], [24], [26], [27].…”

Section: A Offline Runtime Predictionmentioning

confidence: 99%

“…Meanwhile, metrics from previous executions of a job are not always available. As a possible remedy for this issue, we previously proposed a system [27] that facilitates the global sharing of context aware runtime models, allowing for runtime prediction based 1 https://github.com/dos-group/enel-experiments on historical executions of a job by different users [23], [24]. Enel assumes a recurring job and thus, the initial profiling cost can be amortized over time.…”

Section: A Offline Runtime Predictionmentioning

confidence: 99%

“…Various solutions have been proposed regarding the optimization of resource configurations. Some methods are designed for specific processing frameworks [13]- [15], others conduct an iterative profiling strategy [8], [16]- [18], and a third line of work builds runtime models for evaluating possible configurations [7], [19]- [24]. However, even with optimized initial resource allocation, the performance of processing jobs can vary greatly due to factors such as data locality, inference between jobs, and failures which impact the performance over time.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Enel: Context-Aware Dynamic Scaling of Distributed Dataflow Jobs using Graph Propagation

Scheinert,

Zhu,

Thamsen

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Distributed dataflow systems like Spark and Flink enable the use of clusters for scalable data analytics. While runtime prediction models can be used to initially select appropriate cluster resources given target runtimes, the actual runtime performance of dataflow jobs depends on several factors and varies over time. Yet, in many situations, dynamic scaling can be used to meet formulated runtime targets despite significant performance variance.This paper presents Enel, a novel dynamic scaling approach that uses message propagation on an attributed graph to model dataflow jobs and, thus, allows for deriving effective rescaling decisions. For this, Enel incorporates descriptive properties that capture the respective execution context, considers statistics from individual dataflow tasks, and propagates predictions through the job graph to eventually find an optimized new scale-out. Our evaluation of Enel with four iterative Spark jobs shows that our approach is able to identify effective rescaling actions, reacting for instance to node failures, and can be reused across different execution contexts.

show abstract

Section: A Offline Runtime Predictionmentioning

confidence: 99%

Section: A Offline Runtime Predictionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Enel: Context-Aware Dynamic Scaling of Distributed Dataflow Jobs using Graph Propagation

Scheinert,

Zhu,

Thamsen

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Other approaches use runtime data to predict the scale-out and runtime behavior of jobs. This data is gained either from dedicated profiling or previous full executions [3], [12], [15]- [20].…”

Section: B Performance Model-basedmentioning

confidence: 99%

“…Many existing approaches iteratively search for suitable cluster configurations [7]- [11]. Several other approaches [3], [12]- [15] build runtime models, which are then used to evaluate possible configurations, including our previous work [16]- [20]. Here, training data for the runtime models is typically generated with dedicated profiling runs on reduced samples of the dataset, or historical runtime data is used.…”

Section: Introductionmentioning

confidence: 99%

C3O: Collaborative Cluster Configuration Optimization for Distributed Data Processing in Public Clouds

Will,

Thamsen,

Scheinert

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Distributed dataflow systems enable data-parallel processing of large datasets on clusters. Public cloud providers offer a large variety and quantity of resources that can be used for such clusters. Yet, selecting appropriate cloud resources for dataflow jobs --that neither lead to bottlenecks nor to low resource utilization --is often challenging, even for expert users such as data engineers.We present C3O, a collaborative system for optimizing data processing cluster configurations in public clouds based on shared historical runtime data. The shared data is utilized for predicting the runtimes of data processing jobs on different possible cluster configurations, using specialized regression models. These models take the diverse execution contexts of different users into account and exhibit mean absolute errors below 3% in our experimental evaluation with 930 unique Spark jobs.

show abstract

On the Potential of Execution Traces for Batch Processing Workload Optimization in Public Clouds

Scheinert¹,

Alamgiralem²,

Bader³

et al. 2021

2021 IEEE International Conference on Big Data (Big Data)

Self Cite

View full text Add to dashboard Cite

With the growing amount of data, data processing workloads and the management of their resource usage becomes increasingly important. Since managing a dedicated infrastructure is in many situations infeasible or uneconomical, users progressively execute their respective workloads in the cloud. As the configuration of workloads and resources is often challenging, various methods have been proposed that either quickly profile towards a good configuration or determine one based on data from previous runs. Still, performance data to train such methods is often lacking and must be costly collected.In this paper, we propose a collaborative approach for sharing anonymized workload execution traces among users, mining them for general patterns, and exploiting clusters of historical workloads for future optimizations. We evaluate our prototype implementation for mining workload execution graphs on a publicly available trace dataset and demonstrate the predictive value of workload clusters determined through traces only.

show abstract

Bellamy: Reusing Performance Models for Distributed Dataflow Jobs Across Contexts

Abstract: There may be differences between this version and the published version. You are advised to consult the publisher's version if you wish to cite from it.

Cited by 17 publications

References 20 publications

Enel: Context-Aware Dynamic Scaling of Distributed Dataflow Jobs using Graph Propagation

Enel: Context-Aware Dynamic Scaling of Distributed Dataflow Jobs using Graph Propagation

C3O: Collaborative Cluster Configuration Optimization for Distributed Data Processing in Public Clouds

On the Potential of Execution Traces for Batch Processing Workload Optimization in Public Clouds

Contact Info

Product

Resources

About