Minimum Makespan Workflow Scheduling for Malleable Jobs with Precedence Constraints and Lifetime Resource Demands

Chen, Chen; Ke, Xiaodi; Zeyl, Timothy; Du, Kaixiang; Sanjabi, Sam; Bergsma, Shane; Pournaghi, Reza; Chen, Chong

doi:10.1109/icdcs.2019.00204

Cited by 5 publications

(4 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These methods could differ in both the problem formulation and the solver. Most commonly, the optimization problem is formulated as a general Linear Programming (LP [9,12]) problem, Mixed Integer Linear Programming (MILP [11]) problem, or a flowbased graph optimization problem such as Min-Cost Max-Flow (MCMF [4]) problem. For example, TetriSched [40] automatically translates the workload resource requests as a MILP problem and solves it to effectively schedule tasks.…”

Section: Job Schedulingmentioning

confidence: 99%

“…Similar to the heuristic based schedulers, these works are not DAG-aware, and they assume predefined resource demands instead of co-optimizing. There are also schedulers that consider task dependencies and allow for resource demands to be malleable [9,12], but they do not consider cost in the optimization and are not heterogeneityaware. Also, for simplicity they neglect workload characteristics and assume all tasks will run faster with more resources.…”

Section: Job Schedulingmentioning

confidence: 99%

See 1 more Smart Citation

Global Optimization of Data Pipelines in Heterogeneous Cloud Environments

Lin¹,

Xu²,

Bramhavar³

et al. 2022

Preprint

View full text Add to dashboard Cite

Modern production data processing and machine learning pipelines on the cloud are critical components for many cloudbased companies. These pipelines are typically composed of complex workflows represented by directed acyclic graphs (DAGs). Cloud environments are attractive to these workflows due to the wide range of choice with heterogeneous instances and prices that can provide the flexibility for different cost-performance needs. However, this flexibility also leads to the complexity of selecting the right resource configuration (e.g., instance type, resource demands) for each task in the DAG, while simultaneously scheduling the tasks with the selected resources to reach the optimal end-to-end performance and cost. These two decisions are often codependent resulting in an NP-hard scheduling optimization bottleneck. Existing solutions only focus solely on either problem and ignore the co-effect on the end-to-end optimum. We propose AGORA, a scheduler that considers both task-level resource allocation and execution for DAG workflows as a whole in heterogeneous cloud environments. AGORA first (1) studies the characteristics of the tasks from prior runs and gives predictions on resource configurations, and (2) automatically finds the best configuration with its corresponding schedules for the entire workflow with a costperformance objective. We evaluate AGORA in a heterogeneous Amazon Web Services (AWS) cloud environment with multi-tenant workflows served by Airflow and demonstrate a performance improvement up to 45% and cost reduction up to 77% compared to state-of-the-art schedulers. In addition, we apply AGORA to a real-world production trace from Alibaba and show cost reduction of 65% and DAG completion time reduction of 57%. Index Analysis SentimentAnalysis Airline Delay Movie Rec.

show abstract

Section: Job Schedulingmentioning

confidence: 99%

Section: Job Schedulingmentioning

confidence: 99%

Global Optimization of Data Pipelines in Heterogeneous Cloud Environments

Lin¹,

Xu²,

Bramhavar³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…While the task assignment to the servers is not concerned. Chen et al [37] proposed BoltSched scheduling method, which heuristically scheduled tasks iteratively generating the shortest makespan so far, to allocate the amount of resource unit to each task. This work did not consider the resource provisioning problem of underlying resources.…”

Section: Related Workmentioning

confidence: 99%

A Study on Heuristic Task Scheduling Optimizing Task Deadline Violations in Heterogeneous Computational Environments

et al. 2020

View full text Add to dashboard Cite

In this paper, we focus on the problem of optimizing deadline violations for executing tasks in various heterogeneous computational environments. To address the problem, we formulated it as a binary nonlinear programming (BNP) model, which maximize the number of completed tasks and optimize the resource utilization of servers. To solve the BNP model in a polynomial complexity, we propose a heuristic task scheduling method, which iteratively schedules a task to the first core such that the accumulated slack time of all scheduled tasks is minimum, until the core cannot finish any task, and executes tasks with the earliest deadline first in each core to execute as many task as possible in a core. Experiment results based on a real world trace show that our method has upto 100% less task violations, and has the best performance in resource efficiency optimization in overall, compared with eight classical and state-of-the-art heuristic methods.

show abstract

“…There exist many DDL training frameworks [13]- [18] for CPU or GPU clusters, which allow users to share the resources and run their jobs concurrently. Some traditional schedulers [19]- [22] are not specifically designed for DDL training jobs and cannot leverage the characteristics of DDL (such as iterativeness and convergence properties) for maximal training efficiency. Some recent studies [23]- [25] focus on resource provisioning for multiple DDL jobs on the cloud.…”

Section: Introductionmentioning

confidence: 99%

Communication Contention Aware Scheduling of Multiple Deep Learning Training Jobs

Wang¹,

Shi²,

Wang³

et al. 2020

Preprint

View full text Add to dashboard Cite

Distributed Deep Learning (DDL) has rapidly grown its popularity since it helps boost the training performance on high-performance GPU clusters. Efficient job scheduling is indispensable to maximize the overall performance of the cluster when training multiple jobs simultaneously. However, existing schedulers do not consider the communication contention of multiple communication tasks from different distributed training jobs, which could deteriorate the system performance and prolong the job completion time. In this paper, we first establish a new DDL job scheduling framework which organizes DDL jobs as Directed Acyclic Graphs (DAGs) and considers communication contention between nodes. We then propose an efficient algorithm, LWF-κ, to balance the GPU utilization and consolidate the allocated GPUs for each job. When scheduling those communication tasks, we observe that neither avoiding all the contention nor blindly accepting them is optimal to minimize the job completion time. We thus propose a provable algorithm, AdaDUAL, to efficiently schedule those communication tasks. Based on AdaDUAL, we finally propose Ada-SRSF for the DDL job scheduling problem. Simulations on a 64-GPU cluster connected with 10 Gbps Ethernet show that LWF-κ achieves up to 1.59× improvement over the classical first-fit algorithms. More importantly, Ada-SRSF reduces the average job completion time by 20.1% and 36.7%, as compared to the SRSF(1) scheme (avoiding all the contention) and the SRSF(2) scheme (blindly accepting all of two-way communication contention) respectively.

show abstract

Minimum Makespan Workflow Scheduling for Malleable Jobs with Precedence Constraints and Lifetime Resource Demands

Cited by 5 publications

References 21 publications

Global Optimization of Data Pipelines in Heterogeneous Cloud Environments

Global Optimization of Data Pipelines in Heterogeneous Cloud Environments

A Study on Heuristic Task Scheduling Optimizing Task Deadline Violations in Heterogeneous Computational Environments

Communication Contention Aware Scheduling of Multiple Deep Learning Training Jobs

Contact Info

Product

Resources

About