Learning Efficient Co-locations for Scheduling Distributed Dataflows in Shared Clusters

Abstract

Resource management systems like YARN or Mesos allow sharing cluster resources by running dataparallel processing jobs in temporarily reserved containers. Containers, in this context, are logical leases of resources as, for instance, a number of cores and main memory, allocated on a particular node. Typically, containers are used without resource isolation to achieve high degrees of overall resource utilization despite the often fluctuating resource usage of single analytic jobs. However, some combinations of jobs utilize the resources better and interfere less with each other when running on the same nodes than others. This paper presents an approach for improving the resource utilization and job throughput when scheduling recurring distributed data-parallel processing jobs in shared cluster environments. Using a reinforcement learning algorithm, the scheduler continuously learns which jobs are best executed simultaneously on the cluster. We evaluated a prototype implementation of our approach with Hadoop YARN, exemplary Flink jobs from different application domains, and a cluster of commodity nodes. Even though the measure we use to assess the goodness of schedules can still be improved, the results of our evaluation show that our approach increases resource utilization and job throughput.

Mary, Hugo, and Hugo*: Learning to schedule distributed data‐parallel processing jobs on shared clusters

“…While the resource‐aware scheduler shows a more even resource usage distribution also for Queue 1, there is not enough room left for improvement of the overall runtime. Details about this effect, including resource usage graphs for the different resources can be found in the original publication about Mary 2 …”

“…We acknowledge the co‐authors of our previous publications on this topic, 1‐3 especially Benjamin Rabier, Ilya Verbitskiy, and Florian Schmidt. This study was funded by German Ministry for Education and Research (BMBF) as BBDC (01IS14013A and 01IS18025A).…”

“…These systems have become popular tools for workloads that range from data aggregation and search to relational queries, graph processing, and machine learning 7‐9 . Jobs from these diverse domains stress different resources, while the resource demands typically also fluctuate significantly over the runtime of jobs 2,10‐12 . Therefore, multiple jobs usually share cluster resources without isolation, so they can benefit from statistical multiplexing 2,13,14 .…”

“…Jobs from these diverse domains stress different resources, while the resource demands typically also fluctuate significantly over the runtime of jobs 2,10‐12 . Therefore, multiple jobs usually share cluster resources without isolation, so they can benefit from statistical multiplexing 2,13,14 . This is implemented by using resource management systems like YARN 15 and Mesos, 16 which allow users to reserve fractions of cluster resources via the notion of containers, in which users then run one or multiple jobs using the frameworks of their choice.…”

“…This is an extended discussion of the works we published at the Big Data Congress 2017 1 (https://doi.org/10.1109/BigDataCongress.2017.28, © IEEE, 2017), in the STBD journal 4(1) 2 (https://doi.org/10.29268/stbd.2017.4.1.3), and the ParaMo workshop at Euro‐Par 2019 3 (to appear, © Springer, 2019), presenting all our scheduler variants together in one article for the first time. In comparison to our previous publications, we also added a new motivation and a comparison of the related work.…”