Aryl: An Elastic Cluster Scheduler for Deep Learning

Abstract

Companies build separate training and inference GPU clusters for deep learning, and use separate schedulers to manage them. This leads to problems for both training and inference: inference clusters have low GPU utilization when the traffic load is low; training jobs often experience long queueing time due to lack of resources. We introduce Aryl, a new cluster scheduler to address these problems. Aryl introduces capacity loaning to loan idle inference GPU servers for training jobs. It further exploits elastic scaling that scales a training job's GPU allocation to better utilize loaned resources. Capacity loaning and elastic scaling create new challenges to cluster management. When the loaned servers need to be returned, we need to minimize the number of job preemptions; when more GPUs become available, we need to allocate them to elastic jobs and minimize the job completion time (JCT). Aryl addresses these combinatorial problems using principled heuristics. It introduces the notion of server preemption cost which it greedily reduces during server reclaiming. It further relies on the JCT reduction value defined for each additional worker for an elastic job to solve the scheduling problem as a multiple-choice knapsack problem. Prototype implementation on a 64-GPU testbed and large-scale simulation with 15-day traces of over 50,000 production jobs show that Aryl brings 1.53x and 1.50x reductions in average queuing time and JCT, and improves cluster usage by up to 26.9% over the cluster scheduler without capacity loaning or elastic scaling.

Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templates

“…Several recent works have focused on fault-tolerant dataparallel training via dynamically changing the global batch size [16,21,34,52]. Fault-tolerant hybrid-parallel training is more challenging because the model is distributed across multiple GPUs.…”

“…CoDDL [16] balances resource efficiency and short job priority in elastic resource sharing problems. Aryl [21] enables elastic resource sharing between inference and training workloads. Pollux [35] considers both resource utilization and statistical efficiency of training jobs when adaptively allocating resources.…”

“…Model state redundancy in data-parallel training is the only form of "free" redundancy, because each worker already has a copy of the model states. Most solutions harness the inherent redundancy provided by data parallelism and utilize its embarrassingly parallel nature to elastically change the number of GPUs while dynamically changing the global batch size [16,21,34,54]. However, they are unable to extend these benefits to hybrid parallelism and are limited only to dataparallel training.…”