Online Partial Throughput Maximization for Multidimensional Coflow

Im, Sungjin; Shadloo, Maryam; Zheng, Zizhan

doi:10.1109/infocom.2018.8486323

Cited by 7 publications

(4 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These parameters are consumed at different times by each worker depending on the training model. A similar situation arises in web searching [13]. Results with higher ranks can be returned to the user more quickly to improve the user's experience.…”

Section: Introductionmentioning

confidence: 72%

“…Tian et al argue that there are dependencies among coflows in the context of multistage jobs and propose an approximation algorithm [29]. Im et al seek to maximize the partial throughput of coflows, since partially processed coflows could still be useful [13]. [30] explores how to improve the performance of coflow scheduling where both deadline coflows and nondeadline coflows coexist.…”

Section: B Finding New Situationsmentioning

confidence: 99%

“…As in many previous network scheduler designs [13], [16], [29], [31], we abstract out the whole datacenter fabric into a non-blocking giant switch, as shown in Fig. 1(a).…”

Section: A Network Modelmentioning

confidence: 99%

“…Many applications, such as distributed training of deep learning [11], [12] and web searching [13], involve a complex interplay of communication and computation at the participating machines within a stage. The stage usually consists of multiple tasks that are distributed across machines.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Application-Oriented Network Scheduling With Metaflow

et al. 2019

View full text Add to dashboard Cite

Distributed applications usually feature a set of correlated flows between two consecutive computation stages. The scheduling of these flows has a crucial influence on job completion time. Coflow improves performance by optimizing the finish time of the entire set of flows. However, the flows and computing tasks in one application have more complex relationships that exceed the coflow's barrier assumption. In this context, scheduling via coflow abstraction may hurt application performance. Accordingly, we propose metaflow, a traffic abstraction derived from the computation graph of the application. Metaflow reveals the detailed flow requirements of the application and makes it easier to reduce the job completion time. Based on the metaflow, we first develop a mathematical model and formulate the scheduling problem as an integer linear programming (ILP) problem. We further prove that it has an equivalent linear programming (LP) problem through rigorous theoretical analysis in order to solve this ILP problem efficiently. To demonstrate the effectiveness of scheduling with metaflow, we have conducted extensive simulations with both synthetic single jobs and production traces containing multiple jobs. The simulation results verify that our new scheduler adapts well to different jobs and can achieve a significant increase in an average speed of 2.87× on a real-life workload, compared to the state-of-the-art coflow scheduler.INDEX TERMS Datacenter networking, distributed applications, network scheduling.

show abstract

Section: Introductionmentioning

confidence: 72%

Section: B Finding New Situationsmentioning

confidence: 99%

“…As in many previous network scheduler designs [13], [16], [29], [31], we abstract out the whole datacenter fabric into a non-blocking giant switch, as shown in Fig. 1(a).…”

Section: A Network Modelmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations