Scheduling dependent coflows to minimize the total weighted job completion time in datacenters

Tian, Bingchuan; Tian, Chen; Wang, Bingquan; Li, Bo; He, Ze-Hao; Dai, Haipeng; Liu, Kexin; Dou, Wanchun; Chen, Guihai

doi:10.1016/j.comnet.2019.05.010

Cited by 8 publications

(20 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the multi-stage job scenarios, minimizing the average CCT does not necessarily lead to minimizing jobs completion time because the dependencies in a job should be considered. To tackle this problem, several heuristics [12] and approximation solutions [13], [14] have recently been proposed to minimize JCT. However, they simplified the problem with some relaxation and ignored the workload characteristics.…”

Section: Input Datamentioning

confidence: 99%

“…In a multi-stage job scenario, due to the dependency between coflows, minimizing CCT may not minimize JCT. To the best of our knowledge, only a few works [12], [13], [1], [25] considered the coflow scheduling problem in the case of multi-stage jobs. Aalo [12] is the first effective heuristic algorithm with the objective of minimizing JCT, which discusses coflow scheduling in multi-stage jobs, and proposes to prioritize coflows according to dependency orders to schedule coflow.…”

Section: Related Workmentioning

confidence: 99%

“…However, it only takes a short section to discuss the heuristic algorithm without neither formal formulation and analysis. Tian et al [13] proposed a deterministic algorithm with an M-approximation ratio by relaxing the multi-stage job scheduling problem to LP, where M represents the number of machines. Unfortunately, it cannot be used in online cases as all job statistics are required to be known in advance to solve the relevant LP.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

A Scalable Deep Reinforcement Learning Model for Online Scheduling Coflows of Multi-Stage Jobs for High Performance Computing

Wang¹,

Shen²

2021

Preprint

View full text Add to dashboard Cite

Coflow is a recently proposed networking abstraction to help improve the communication performance of data-parallel computing jobs. In multi-stage jobs, each job consists of multiple coflows and is represented by a Directed Acyclic Graph (DAG). Efficiently scheduling coflows is critical to improve the data-parallel computing performance in data centers. Compared with hand-tuned scheduling heuristics, existing work DeepWeave [1] utilizes Reinforcement Learning (RL) framework to generate highly-efficient coflow scheduling policies automatically. It employs a graph neural network (GNN) to encode the job information in a set of embedding vectors, and feeds a flat embedding vector containing the whole job information to the policy network. However, this method has poor scalability as it is unable to cope with jobs represented by DAGs of arbitrary sizes and shapes, which requires a large policy network for processing a high-dimensional embedding vector that is difficult to train. In this paper, we first utilize a directed acyclic graph neural network (DAGNN) to process the input and propose a novel Pipelined-DAGNN, which can effectively speed up the feature extraction process of the DAGNN. Next, we feed the embedding sequence composed of schedulable coflows instead of a flat embedding of all coflows to the policy network, and output a priority sequence, which makes the size of the policy network depend on only the dimension of features instead of the product of dimension and number of nodes in the job's DAG. Furthermore, to improve the accuracy of the priority scheduling policy, we incorporate the Self-Attention Mechanism into a deep RL model to capture the interaction between different parts of the embedding sequence to make the output priority scores relevant. Based on this model, we then develop a coflow scheduling algorithm for online multi-stage jobs. Our simulation results are based on the real trace of Facebook. Compared with a state-of-the-art approach, our model can shorten the average weighted job completion time by up to 40.42% and complete jobs at least 1.68 times faster. It also has better scalability and robustness.

show abstract

Section: Input Datamentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

A Scalable Deep Reinforcement Learning Model for Online Scheduling Coflows of Multi-Stage Jobs for High Performance Computing

Wang¹,

Shen²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…For jobs with a single communication stage, minimizing the average completion times of coflows results in the job's latency improvement. However, for multi-stage jobs, minimizing the average coflow completion time might not be the right metric and might even lead to a worse performance, as it ignores the dependencies between coflows in a job [5], [6], [11].…”

Section: Introductionmentioning

confidence: 99%

“…Each job is represented by a DAG (Directed Acyclic Graph) among its coflows that capture the (Starts-After) dependencies among the coflows. As in [5], [8]- [11], the data center network is modeled as an m×m switch where m is the number of servers (see Section II for the formal job and data center network model). As an illustration, Figure 1 shows one multi-stage job in a 2 × 2 switch.…”

Section: Introductionmentioning

confidence: 99%

Scheduling Coflows with Dependency Graph

Shafiee¹,

Ghaderi²

2020

Preprint

View full text Add to dashboard Cite

Applications in data-parallel computing typically consist of multiple stages. In each stage, a set of intermediate parallel data flows (Coflow) is produced and transferred between servers to enable starting of next stage. While there has been much research on scheduling isolated coflows, the dependency between coflows in multi-stage jobs has been largely ignored. In this paper, we consider scheduling coflows of multi-stage jobs represented by general DAGs (Directed Acyclic Graphs) in a shared data center network, so as to minimize the total weighted completion time of jobs. This problem is significantly more challenging than the traditional coflow scheduling, as scheduling even a single multi-stage job to minimize its completion time is shown to be NP-hard. In this paper, we propose a polynomial-time algorithm with approximation ratio of O(µ log(m)/ log(log(m))), where µ is the maximum number of coflows in a job and m is the number of servers. For the special case that the jobs' underlying dependency graphs are rooted trees, we modify the algorithm and improve its approximation ratio. To verify the performance of our algorithms, we present simulation results using real traffic traces that show up to 53% improvement over the prior approach. We conclude the paper by providing a result concerning an optimality gap for scheduling coflows with general DAGs.

show abstract

Fast and isolation guaranteed coflow scheduling via traffic forecasting in multi-tenant environment

Li,

Zhang,

Yang

et al. 2024

J Supercomput

View full text Add to dashboard Cite

Scheduling dependent coflows to minimize the total weighted job completion time in datacenters

Cited by 8 publications

References 13 publications

A Scalable Deep Reinforcement Learning Model for Online Scheduling Coflows of Multi-Stage Jobs for High Performance Computing

A Scalable Deep Reinforcement Learning Model for Online Scheduling Coflows of Multi-Stage Jobs for High Performance Computing

Scheduling Coflows with Dependency Graph

Fast and isolation guaranteed coflow scheduling via traffic forecasting in multi-tenant environment

Contact Info

Product

Resources

About