On the Potential of Execution Traces for Batch Processing Workload Optimization in Public Clouds

Abstract

With the growing amount of data, data processing workloads and the management of their resource usage becomes increasingly important. Since managing a dedicated infrastructure is in many situations infeasible or uneconomical, users progressively execute their respective workloads in the cloud. As the configuration of workloads and resources is often challenging, various methods have been proposed that either quickly profile towards a good configuration or determine one based on data from previous runs. Still, performance data to train such methods is often lacking and must be costly collected.In this paper, we propose a collaborative approach for sharing anonymized workload execution traces among users, mining them for general patterns, and exploiting clusters of historical workloads for future optimizations. We evaluate our prototype implementation for mining workload execution graphs on a publicly available trace dataset and demonstrate the predictive value of workload clusters determined through traces only.

Collaborative Cluster Configuration for Distributed Data-Parallel Processing: A Research Overview

“…In several of our prior works [16,31,33,32,20], we discussed the idea of exploiting similarities between different jobs and their executions, cultivating runtime data in a collaborative manner among numerous users and thereby improving the prediction capabilities of individual users. This includes decentralized system architectures for sharing context-aware runtime metrics, as well as similarity matching between jobs.…”

Lotaru: Locally Estimating Runtimes of Scientific Workflow Tasks in Heterogeneous Clusters

“…Both approaches have in common that they need at least comprehensive knowledge about execution times of all tasks on all available nodes. However, these values are not available in advance but must be determined either by asking users for estimates [18,22,23], by analyzing historical traces [35,36,42], or by using some form of online learning [43,45]. Lotaru aims to estimate the runtime for all task-node pairs in a cluster to enable the use of existing scheduling methods in real-world systems.…”

Ruya: Memory-Aware Iterative Optimization of Cluster Configurations for Big Data Processing

“…Some approaches use runtime data to predict the job's scale-out and runtime behavior. This data is gained either from dedicated profiling or previous full executions [7], [25]- [31]. The models can then be used to predict the execution performance for different cluster configurations, and the most resource-efficient one will be chosen.…”