Near-Data Scheduling for Data Centers with Multiple Levels of Data Locality

Yekkehkhany, Ali

doi:10.48550/arxiv.1702.07802

Cited by 4 publications

(8 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…is signi cant improvement over JSQ-MaxWeight algorithm shows that JSQ-MaxWeight is not delay optimal and supports the possibility that the GB-PANDAS algorithm is delay optimal in a larger region than the JSQ-MaxWeight algorithm. By the intuition we got from the delay optimality proof of the JSQ-MaxWeight algorithm for two locality levels in [32], [37], [30], and [34], we simulated the system under a load for which we believe JSQ-MaxWeight is delay optimal. Figure 5 shows the result for this speci c load and we see that both the GB-PANDAS and JSQ-MaxWeight algorithms have the same performance at high loads, which again supports our guess on delay optimality of our proposed algorithm.…”

Section: Simulation Resultsmentioning

confidence: 99%

GB-PANDAS: Throughput and heavy-traffic optimality analysis for affinity scheduling

Yekkehkhany,

Hojjati,

Hajiesmaili

2017

Preprint

Self Cite

View full text Add to dashboard Cite

Dynamic a nity scheduling has been an open problem for nearly three decades. e problem is to dynamically schedule multi-type tasks to multi-skilled servers such that the resulting queueing system is both stable in the capacity region (throughput optimality) and the mean delay of tasks is minimized at high loads near the boundary of the capacity region (heavy-tra c optimality). As for applications, data-intensive analytics like MapReduce, Hadoop, and Dryad t into this se ing, where the set of servers is heterogeneous for di erent task types, so the pair of task type and server determines the processing rate of the task. e load balancing algorithm used in such frameworks is an example of a nity scheduling which is desired to be both robust and delay optimal at high loads when hot-spots occur. Fluid model planning, the MaxWeight algorithm, and the generalized cµ-rule are among the rst algorithms proposed for a nity scheduling that have theoretical guarantees on being optimal in di erent senses, which will be discussed in the related work section. All these algorithms are not practical for use in data center applications because of their non-realistic assumptions. e join-the-shortest-queue-MaxWeight (JSQ-MaxWeight), JSQ-Priority, and weighted-workload algorithms are examples of load balancing policies for systems with two and three levels of data locality with a rack structure. In this work, we propose the Generalized-Balanced-Pandas algorithm (GB-PANDAS) for a system with multiple levels of data locality and prove its throughput optimality. We prove this result under an arbitrary distribution for service times, whereas most previous theoretical work assumes geometric distribution for service times.e extensive simulation results show that the GB-PANDAS algorithm alleviates the mean delay and has a be er performance than the JSQ-MaxWeight algorithm by up to twofold at high loads. We believe that the GB-PANDAS algorithm is heavy-tra c optimal in a larger region than JSQ-MaxWeight, which is an interesting problem for future work.

show abstract

Section: Simulation Resultsmentioning

confidence: 99%

GB-PANDAS: Throughput and heavy-traffic optimality analysis for affinity scheduling

Yekkehkhany,

Hojjati,

Hajiesmaili

2017

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…One could consider developing a P2MP joint scheduling and routing scheme with throughput-optimality as the objective. However, in general, throughput-optimality does not necessarily lead to highest performance (e.g., lowest latency) [54], [55].…”

Section: Related Workmentioning

confidence: 99%

QuickCast: Fast and Efficient Inter-Datacenter Transfers using Forwarding Tree Cohorts

Noormohammadpour¹,

Raghavendra²,

Kandula³

et al. 2018

Preprint

View full text Add to dashboard Cite

Large inter-datacenter transfers are crucial for cloud service efficiency and are increasingly used by organizations that have dedicated wide area networks between datacenters. A recent work uses multicast forwarding trees to reduce the bandwidth needs and improve completion times of point-tomultipoint transfers. Using a single forwarding tree per transfer, however, leads to poor performance because the slowest receiver dictates the completion time for all receivers. Using multiple forwarding trees per transfer alleviates this concern-the average receiver could finish early; however, if done naively, bandwidth usage would also increase and it is apriori unclear how best to partition receivers, how to construct the multiple trees and how to determine the rate and schedule of flows on these trees. This paper presents QuickCast, a first solution to these problems. Using simulations on real-world network topologies, we see that QuickCast can speed up the average receiver's completion time by as much as 10× while only using 1.04× more bandwidth; further, the completion time for all receivers also improves by as much as 1.6× faster at high loads.

show abstract

“…In fact, λL ,m is the decomposition of λL. Assuming that a server can afford at most load 1 for all local, rack-local, and remote tasks, the capacity region can be characterized as follows [12], [13]:…”

Section: A Capacity Region Realizationmentioning

confidence: 99%

Affinity Scheduling and the Applications on Data Center Scheduling with Data Locality

Kavousi¹

2017

Preprint

View full text Add to dashboard Cite

MapReduce framework is the de facto standard in Hadoop. Considering the data locality in data centers, the load balancing problem of map tasks is a special case of affinity scheduling problem. There is a huge body of work on affinity scheduling, proposing heuristic algorithms which try to increase data locality in data centers like Delay Scheduling and Quincy. However, not enough attention has been put on theoretical guarantees on throughput and delay optimality of such algorithms. In this work, we present and compare different algorithms and discuss their shortcoming and strengths. To the best of our knowledge, most data centers are using static load balancing algorithms which are not efficient in any ways and results in wasting the resources and causing unnecessary delays for users.

show abstract

Near-Data Scheduling for Data Centers with Multiple Levels of Data Locality

Cited by 4 publications

References 0 publications

GB-PANDAS: Throughput and heavy-traffic optimality analysis for affinity scheduling

GB-PANDAS: Throughput and heavy-traffic optimality analysis for affinity scheduling

QuickCast: Fast and Efficient Inter-Datacenter Transfers using Forwarding Tree Cohorts

Affinity Scheduling and the Applications on Data Center Scheduling with Data Locality

Contact Info

Product

Resources

About