Decentralized task-aware scheduling for data center networks

Abstract

Many data center applications perform rich and complex tasks (e.g., executing a search query or generating a user's news-feed). From a network perspective, these tasks typically comprise multiple flows, which traverse different parts of the network at potentially different times. Most network resource allocation schemes, however, treat all these flows in isolation -rather than as part of a task -and therefore only optimize flow-level metrics.In this paper, we show that task-aware network scheduling, which groups flows of a task and schedules them together, can reduce both the average as well as tail completion time for typical data center applications. To achieve these benefits in practice, we design and implement Baraat, a decentralized task-aware scheduling system. Baraat schedules tasks in a FIFO order but avoids head-of-line blocking by dynamically changing the level of multiplexing in the network. Through experiments with Memcached on a small testbed and large-scale simulations, we show that Baraat outperforms state-of-the-art decentralized schemes (e.g., pFabric) as well as centralized schedulers (e.g., Orchestra) for a wide range of workloads (e.g., search, analytics, etc).

IoNCloud: Exploring application affinity to improve utilization and predictability in datacenters

“…EyeQ [12] and Gatekeeper [18], in turn, can offer bandwidth guarantees only when the core of the network is congestion-free. Baraat [1] and Varys [10] achieve high network utilization, but cannot provide strict bandwidth guarantees for tenants. Finally, ElasticSwitch [8] and the Logistic Model [3] are orthogonal to our approach, as they assume there exists an allocation method in the cloud platform (i.e., applications are already allocated).…”

Minimizing average coflow completion time with decentralized scheduling

“…In these frameworks, data-intensive jobs are divided into multiple successive data-parallel computation stages; and a succeeding computation stage cannot start until getting all its required inputs, which is exactly the outputs of the previous stage. Furthermore, the transmission of the intermediate data is not a negligible phase in a job [1]- [3]. For example, some real traces from Facebook show that, the data transferring phase between successive stages accounts for 33% of the running times of jobs in the system [1].…”

“…For example, some real traces from Facebook show that, the data transferring phase between successive stages accounts for 33% of the running times of jobs in the system [1]. Accordingly, speed up the data transfer between computation stages will accelerate the job completion and increase the data center utilization [1]- [3].…”

“…Many existing works [1]- [3] focus on minimizing average CCT in DCNs. To the best of our knowledge, Varys [2] and Baraat [3] are the state-of-the-art schemes in centralized and decentralized manner, respectively.…”

“…To the best of our knowledge, Varys [2] and Baraat [3] are the state-of-the-art schemes in centralized and decentralized manner, respectively. However, centralized schemes like Varys have the scalability problem.…”