Gemini: An Adaptive Performance-Fairness Scheduler for Data-Intensive Cluster Computing

Abstract

In data-intensive cluster computing platforms such as Hadoop YARN, performance and fairness are two important concerns for users. Existing studies show that, because of the resource contention between users/jobs, there is a tradeoff between the performance and fairness. In our work, we observe that such trade-off is related to the resource demand of the workload and is changing with the variation of multi-resource demand of submitted jobs during the computation. We also find that having an algorithm to be aware of the resource demand variation is important for the bi-criteria optimization between performance and fairness. However, most previous studies are not aware of this and design their heuristic algorithms with the assumption of fixed trade-off. In this paper, we propose a adaptive scheduler called Gemini for Hadoop YARN. For Gemini, it first develops a regression approach to construct a model which can estimate the performance improvement and the fairness loss under the sharing computation compared to the exclusive non-sharing scenario. Next, it leverages the model to guide the resource allocation for pending tasks to optimize the performance of the cluster given the user-defined fairness level. Instead of using a static scheduling policy, Gemini adaptively decides the proper scheduling policy according to the current running workload. We implement Gemini in Hadoop YARN. Experimental results show that Gemini outperforms the state-ofthe-art work in two aspects. 1) For the same fairness loss, Gemini increases the performance improvement up to 225% and 200% in real deployment and the large-scale simulation, respectively; 2) For the same performance improvement, Gemini reduces the fairness loss up to 70% and 62.5% in real deployment and the large-scale simulation, respectively.

Hugo: A Cluster Scheduler that Efficiently Learns to Select Complementary Data-Parallel Jobs

“…Gemini [16] uses a model that captures the tradeoff between performance improvement and fairness loss for jobs scheduled in shared clusters. The model quantifies the complementarity in the resource demands of jobs and is trained on historic workload data.…”

“…These systems have become popular tools for workloads that range from data aggregation and search to relational queries, graph processing, and machine learning [17,9,8,15]. Jobs from these diverse domains stress different resources, while the resource demands typically also fluctuate significantly over the runtime of jobs [18,21,16]. Therefore, multiple jobs usually share cluster resources without isolation, so they can benefit from statistical multiplexing [26,20,23].…”

Mary, Hugo, and Hugo*: Learning to schedule distributed data‐parallel processing jobs on shared clusters

“…Gemini 11 uses a model that captures the tradeoff between performance improvement and fairness loss for jobs scheduled in shared clusters. The model quantifies the complementarity in the resource demands of jobs and is trained on historic workload data.…”

“…These systems have become popular tools for workloads that range from data aggregation and search to relational queries, graph processing, and machine learning 7‐9 . Jobs from these diverse domains stress different resources, while the resource demands typically also fluctuate significantly over the runtime of jobs 2,10‐12 . Therefore, multiple jobs usually share cluster resources without isolation, so they can benefit from statistical multiplexing 2,13,14 .…”

“…Therefore, schedulers should actively co‐locate those jobs that share resources efficiently. The benefits of such approaches have been demonstrated before with multiple schedulers that explicitly take combined resource utilization and interference among co‐located workloads into account 11,18,27 or learn the impact of this indirectly, 28,29 taking advantage of the recurrence of a majority of jobs in industry workloads 21,26 …”