Hierarchical scheduling for diverse datacenter workloads

Abstract

There has been a recent industrial effort to develop multi-resource hierarchical schedulers. However, the existing implementations have some shortcomings in that they might leave resources unallocated or starve certain jobs. This is because the multi-resource setting introduces new challenges for hierarchical scheduling policies. We provide an algorithm, which we implement in Hadoop, that generalizes the most commonly used multi-resource scheduler, DRF [1], to support hierarchies. Our evaluation shows that our proposed algorithm, H-DRF, avoids the starvation and resource inefficiencies of the existing open-source schedulers and outperforms slot scheduling.

Virtual machine priority adaption to enforce fairness among cloud users

“…The most prominent definition of data center fairness is Dominant Resource Fairness (DRF) as introduced in [9]. DRF has been extended in many directions, for example by indivisibilities of resources and user hierarchies [10], [11], [12], [13]. A third approach to data centers fairness is the extension of Proportional Fairness [14] to multiple resources [15], [16].…”

Strategyproof allocation of discrete jobs on multiple machines

“…Later, Hadoop MapReduce implemented the DRF model for their Capacity ([apache.org 2014a]) and Fair Schedulers ([apache.org 2014b]). It has also been extended to be used in real-time to do finegrained packet scheduling inside routers ( [Ghodsi et al 2012]) as well as hierarchical scheduling within large organizations ( [Bhattacharya et al 2013]). Many of its more theoretical properties have been studied, e.g.…”

“…For the internal, the introduction of the dominant resource fairness ( ) protocol has spurred much work on strategyproof allocation for Leontief Economies ( [Parkes et al 2012], [Dolev et al 2012], [Li and Xue 2013]) leading to important insights and unexpected connections between computer science ([Joe-Wong et al 2012]) and economics ( [Friedman et al 2011]). DRF has been extended to be used inside routers ( [Ghodsi et al 2012]) as well as large organizations with hierarchies ( [Bhattacharya et al 2013]). In addition, it has been deployed in production, running on thousands of nodes at Twitter in the Mesos resource manager ( ).…”

Dominant Fairness Fairness: Hierarchical Scheduling for Multiple Resources in Heterogeneous Datacenters

“…Each node i has a weight w i , which is a basis for obtaining resources, i.e., node n 3 should get w 3 / 4 i=1 w i resources from its parent node n r , and node n 3,1 should get w 3,1 / 2 i=1 w 3,i resources from n 3 . Because DRF has some shortcomings with respect to hierarchical scheduling, Arka et al proposed Hierarchy DRF (H-DRF) [1], which can deal with the issues in DRF such as resources left unallocated or the starving of certain jobs.…”

“…However, the latest scheduling technique, Hierarchy Dominant Resource Fairness (H-DRF) [1], has some shortcomings in heterogeneous environments, such as starving certain jobs or unfair resource allocation. This is because a heterogeneous environment brings new challenges.…”