Achieving Efficient Distributed Scheduling with Message Queues in the Cloud for Many-Task Computing and High-Performance Computing

Abstract

Abstract-Task scheduling and execution over large scale, distributed systems plays an important role on achieving good performance and high system utilization. Due to the explosion of parallelism found in today's hardware, applications need to perform over-decomposition to deliver good performance; this over-decomposition is driving job management systems' requirements to support applications with a growing number of tasks with finer granularity. Our goal in this work is to provide a compact, light-weight, scalable, and distributed task execution framework (CloudKon) that builds upon cloud computing building blocks (Amazon EC2, SQS, and DynamoDB). Most of today's state-of-the-art job execution systems have predominantly Master/Slaves architectures, which have inherent limitations, such as scalability issues at extreme scales and single point of failures. On the other hand distributed job management systems are complex, and employ non-trivial load balancing algorithms to maintain good utilization. CloudKon is a distributed job management system that can support both HPC and MTC workloads with millions of tasks/jobs. We compare our work with other state-of-the-art job management systems including Sparrow and MATRIX. The results show that CloudKon delivers better scalability compared to other state-of-the-art systems for some metricsall with a significantly smaller code-base (5%).

Optimizing load balancing and data-locality with data-aware scheduling

“…This mechanism suffers longtail problem under heterogeneous workloads [34] due to early binding of tasks to worker resources. We have compared Sparrow and the basic MATRIX without data-aware scheduling technique using heterogeneous workloads in [39], and MATRIX outperforms Sparrow by 9X. Furthermore, there is an implementation barrier with Sparrow as it is developed in Java, which has little support in high-end computing systems.…”

“…CloudKon [39] has similar architecture as MATRIX, except that CloudKon focuses on the Cloud environment, and relies on the Cloud services, SQS [40] to do distributed load balancing, and DynamoDB [41] as the DKVS to keep task metadata. Relying on the Cloud services could facilitate the easier development, at the cost of potential loss of performance and control.…”

Towards In-Order and Exactly-Once Delivery Using Hierarchical Distributed Message Queues

“…This is all done on atomic ba time a copy of message is locked in the fron the message is successfully delivered to the if router didn't get the message from the will get it from replica node. Compared t our system offers exactly one delivery exactly one delivery functionality in Ama DynamoDB as used in CloudKon, the per Amazon SQS decreases by 30% [8]. W service bus provides At Least Once Proce failure.…”

“…A of repeated message we still will be hav messages. So if we want to stop these re from SQS, we can use DyanmoDB for han delivery of message but it will probab performance of the whole system by 30 CloudKon [8].…”

Deployment Method of Virtual Machine Based on PSO Algorithm Modified by Gauss Strategy

“…In recent years, driven by the rapid development of computer hardware resources, the physical resources (including CPU, network, storage, I / O devices, VMware as the representative) virtualization technology has made considerable progress, and provides a solid foundation for the rapid development of cloud computing [4].…”