Hugo: A Cluster Scheduler that Efficiently Learns to Select Complementary Data-Parallel Jobs

Abstract

Distributed data processing systems like MapReduce, Spark, and Flink are popular tools for analysis of large datasets with cluster resources. Yet, users often overprovision resources for their data processing jobs, while the resource usage of these jobs also typically fluctuates considerably. Therefore, multiple jobs usually get scheduled onto the same shared resources to increase the resource utilization and throughput of clusters. However, job runtimes and the utilization of shared resources can vary significantly depending on the specific combinations of co-located jobs. This paper presents Hugo, a cluster scheduler that continuously learns how efficiently jobs share resources, considering metrics for the resource utilization and interference among co-located jobs. The scheduler combines offline grouping of jobs with online reinforcement learning to provide a scheduling mechanism that efficiently generalizes from specific monitored job combinations yet also adapts to changes in workloads. Our evaluation of a prototype shows that the approach can reduce the runtimes of exemplary Spark jobs on a YARN cluster by up to 12.5%, while resource utilization is increased and waiting times can be bounded.

Mary, Hugo, and Hugo*: Learning to schedule distributed data‐parallel processing jobs on shared clusters

“…In the following we describe the method of grouping jobs, learning good co‐location of groups, and the evaluation of the scheduling decisions by Hugo. More details about the calculation of the scheduling probabilities as well as the prototype implementation can be found in the previous publication about Hugo 3 …”

“…We acknowledge the co‐authors of our previous publications on this topic, 1‐3 especially Benjamin Rabier, Ilya Verbitskiy, and Florian Schmidt. This study was funded by German Ministry for Education and Research (BMBF) as BBDC (01IS14013A and 01IS18025A).…”

“…This is an extended discussion of the works we published at the Big Data Congress 2017 1 (https://doi.org/10.1109/BigDataCongress.2017.28, © IEEE, 2017), in the STBD journal 4(1) 2 (https://doi.org/10.29268/stbd.2017.4.1.3), and the ParaMo workshop at Euro‐Par 2019 3 (to appear, © Springer, 2019), presenting all our scheduler variants together in one article for the first time. In comparison to our previous publications, we also added a new motivation and a comparison of the related work.…”

A HPC Co-scheduler with Reinforcement Learning

“…As mentioned in previous sections, ASA X is a stateful extension of [36], where the main difference is that ASA X can incorporate previous decisions in a RL approach. In [39], the authors combine offline job classification with online RL to improve collocations. This approach can accelerate convergence, although it might have complex consequences when new unknown jobs do not fit into the initial classification.…”

ML-driven classification scheme for dynamic interference-aware resource scheduling in cloud infrastructures

“…In cloud computing ecosystems, consolidating multiple user applications onto multi-core servers generates interference between co-hosted applications, which impacts application performance. To minimize interference effects and improve application performance, a common solution is to utilize schedulers that consider interference issues [26].…”