A Highly Efficient Data Locality Aware Task Scheduler for Cloud-Based Systems

Ru, Jia; Yang, Yun; Grundy, John; Keung, Jacky; Li, Hao

doi:10.1109/cloud.2019.00089

Cited by 9 publications

(5 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the contrary, the naïve DRF was proposed in work Reference 35 and was only compared with slot‐based and CPU‐based fair scheduling algorithm. Similarly, more experiments about DLAforBT presented in Section 6 were described in our work 64 and more experiments about PDSonQueue showed in Section 7 were described in our work 65 …”

Section: Methodsmentioning

confidence: 94%

“…The overall resource allocation strategy used in our scheduling framework was proposed in our earlier work, 24 and the described deadline constrained scheduler and data locality aware scheduler in 3DSF have been extended from our earlier work 64,65 . A major new contribution of this work was to integrate all these three schedulers together, with highly efficient performance.…”

Section: A Deadline‐constrained and Data Locality‐aware Dynamic Schedmentioning

confidence: 99%

“…Scheduling optimization is an important approach to improve data locality by attempting to locate a task and its related data on the same node. We describe a novel optimized data locality aware task scheduler for balancing time consumption and network bandwidth traffic—DLAforBT—to improve data locality for tasks, with the optimal placement policy exhibiting a threshold‐based structure, improving resource utilization and throughput and shortening completion time 64 . DLAforBT 64 transforms the data locality scheduling problem into the well‐known maximum weighted bipartite matching (MWBM) graph problem; uses a judgment mechanism to dynamically adjust task allocation and an embedded precise prediction model to determine moving computation or moving data, which demonstrates the benefits it offers to cloud systems; and ranks a list of idle computation capacity nodes based on descending order and gives different priorities to nodes, to maximize resource usage .…”

Section: A Data Locality Aware Task Schedulermentioning

confidence: 99%

“…If tasks T require large amount of data, with the aim of enhancing tasks' data locality ( Line:9 ), the data locality aware scheduler – DLAforBT 64 is invoked to locate tasks to their corresponding nodes to improve data locality ( Lines:10‐12 ). Otherwise, if tasks T are small, the optional data locality aware scheduler – DLAforBT 64 will be skipped and tasks T directly wait for resource allocation. A corresponding container con t with a timestamp is created for task t ( Line:15 ).…”

Section: Integration Of Individual Schedulersmentioning

confidence: 99%

See 3 more Smart Citations

An efficient deadline constrained and data locality aware dynamic scheduling framework for multitenancy clouds

Yang

Grundy

et al. 2020

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

Summary Scheduling and resource allocation in clouds is used to harness the power of the underlying resource pool. Service providers can meet quality of service (QoS) requirements of tenants specified in Service Level Agreements. Improving resource allocation ensures that all tenants will receive fairer access to system resources, which improves overall utilization and throughput. Real‐time applications and services require critical deadlines in order to guarantee QoS. A growing number of data‐intensive applications drive the optimization of scheduling through utilizing data locality in which the scheduler locates a task and ensures the task's relevant data to be on the same server. Choosing suitable scheduling mechanisms for running applications that support multitenancy has consistently been a major challenge. This work proposes a new adaptive Deadline constrained and Data locality aware Dynamic Scheduling Framework “ 3DSF“ that orchestrates different schedulers based on varied requirements. This framework considers tenants' deadline‐based QoS requirements, cloud system's performance and a method of resource allocation to improve resource utilization, system throughput and reduce jobs' completion time. 3DSF contains: (a) a real‐time, preemptive, deadline constrained job scheduler, (b) an optimized data locality aware scheduler, (c) an improved Dominant Resource Fairness greedy resource allocation approach, and (d) an adaptive suite to integrate above‐mentioned schedulers together.

show abstract

Section: Methodsmentioning

confidence: 94%

Section: A Deadline‐constrained and Data Locality‐aware Dynamic Schedmentioning

confidence: 99%

Section: A Data Locality Aware Task Schedulermentioning

confidence: 99%

Section: Integration Of Individual Schedulersmentioning

confidence: 99%

See 2 more Smart Citations

An efficient deadline constrained and data locality aware dynamic scheduling framework for multitenancy clouds

Yang

Grundy

et al. 2020

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

show abstract

“…A more recent research effort proposed a novel scheduler for cloud-based systems that utilises machine learning to decide when scheduling tasks away from their data is less expensive than moving data to achieve the maximum locality (e.g. when the network bandwidth is unstable) [10].…”

Section: Related Workmentioning

confidence: 99%

Distributed Data Locality-Aware Job Allocation

Markovic,

Kolovos,

Soares Indrusiak

2023

Proceedings of the SC '23 Workshops of the International Conference on High Performance Computing, Network, Storage, and Analys

View full text Add to dashboard Cite

Scheduling tasks close to their associated data is crucial in distributed systems to minimize network traffic and latency. Some Big Data frameworks like Apache Spark employ locality functions and job allocation algorithms to minimize network traffic and execution times. However, these frameworks rely on centralized mechanisms, where the master node determines data locality by allocating tasks to available workers with minimal data transfer time, ignoring variances in worker configurations and availability. To address these limitations, we propose a decentralized approach to locality-driven scheduling that grants workers autonomy in the job allocation process while factoring in workers' configurations, such as network and CPU speed differences. Our approach is developed and evaluated on Crossflow, a distributed stream processing platform with data-aware independent worker nodes. Preliminary evaluation experiments indicate that our approach can yield up to 3.57x faster execution times when compared to the baseline centralized approach where the master controls data locality. CCS CONCEPTS• Computing methodologies → Distributed algorithms; • Software and its engineering → Development frameworks and environments.

show abstract

Smart Grid Data Anomaly Detection Method Based on Cloud Computing Platform

Liu

Cao

et al. 2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

A Highly Efficient Data Locality Aware Task Scheduler for Cloud-Based Systems

Cited by 9 publications

References 4 publications

An efficient deadline constrained and data locality aware dynamic scheduling framework for multitenancy clouds

An efficient deadline constrained and data locality aware dynamic scheduling framework for multitenancy clouds

Distributed Data Locality-Aware Job Allocation

Smart Grid Data Anomaly Detection Method Based on Cloud Computing Platform

Contact Info

Product

Resources

About