Multi-level queue for task scheduling in heterogeneous distributed computing system

Biswas, Tarun; Kuila, Pratyay; Ray, Anjan Kumar

doi:10.1109/icaccs.2017.8014673

Cited by 16 publications

(5 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Two mainstream system architectures are available for building parallel and distributed computing systems: HPC and cluster computing. The first one is used to build ultra-HPC systems called supercomputers, which integrate thousands of computing nodes and millions of CPU/GPU cores to perform various operations in parallel [15][16][17][18] to solve computationally intensive computing tasks such as weather forecasting, highenergy physics, genetic analysis, and atmospheric science [19,20] .…”

Section: Clusters and Hpcmentioning

confidence: 99%

Survey of Distributed Computing Frameworks for Supporting Big Data Analysis

Sun

et al. 2023

Big Data Min. Anal.

View full text Add to dashboard Cite

Distributed computing frameworks are the fundamental component of distributed computing systems.They provide an essential way to support the efficient processing of big data on clusters or cloud. The size of big data increases at a pace that is faster than the increase in the big data processing capacity of clusters. Thus, distributed computing frameworks based on the MapReduce computing model are not adequate to support big data analysis tasks which often require running complex analytical algorithms on extremely big data sets in terabytes.In performing such tasks, these frameworks face three challenges: computational inefficiency due to high I/O and communication costs, non-scalability to big data due to memory limit, and limited analytical algorithms because many serial algorithms cannot be implemented in the MapReduce programming model. New distributed computing frameworks need to be developed to conquer these challenges. In this paper, we review MapReduce-type distributed computing frameworks that are currently used in handling big data and discuss their problems when conducting big data analysis. In addition, we present a non-MapReduce distributed computing framework that has the potential to overcome big data analysis challenges.

show abstract

Section: Clusters and Hpcmentioning

confidence: 99%

Survey of Distributed Computing Frameworks for Supporting Big Data Analysis

Sun

et al. 2023

Big Data Min. Anal.

View full text Add to dashboard Cite

show abstract

“…A multi-level queue (MLQ) task planning algorithm was suggested by Biswas et al [12] to reduce the range of parallels between subtasks while infringing precursors relations. Our principal objective is to take advantage of algorithms for the scheduling of algorithm tasks in terms of the span, time complexity, the use of resources, system performance and dynamic nature.…”

Section: Literature Reviewmentioning

confidence: 99%

864 Published By: Blue Eyes Intelligence Engineering & Sciences Publication Retrieval Number: E6080018520 /2020©BEIESP DOI:10.35940/ijrte.E6080.018520 Journal Website: www.ijrte.org Multi Level Queue Scheduling With Particle Swarm Optimization (Mlqs-Pso) Of Vms in Queueing Heterogeneous Cloud Computing Systems

Rekha*¹,

Kalaiselvi²

2020

IJRTE

View full text Add to dashboard Cite

This article investigates in cloud computing systems about problem of delay optimal Virtual Machine (VM) scheduling holds constant resources with full infrastructure like CPU, memory and storage in the resource pool. Cloud computing offers users with VMs as utilities. Cloud consumers randomly demand different VM types over time, and the usual length of the VM hosting differs greatly. A scheduling algorithm for a multilevel queue divides the prepared queue towards lengthy and various queues. System is allocated with single queue in to several longer queues. The systems are allocated to one queue indefinitely, usually on any basis of process property, like memory size, process priority, or process sort. Every queue will have its self-algorithm for scheduling. Likewise, a system that's taking in a less preference queue is so lengthy, a high-priority queue can be transferred. Using Particle Swarm Optimization Algorithm (MQPSO), Multi-level queue scheduling has been done. To evaluate the solutions, it explores both Shortest-Job-First (SJF) buffering and Min-Min Best Fit (MMBF) programming algorithms, i.e., SJF-MMBF. The scheme incorporating the SJF-ELM-specific scheduling algorithms depending SJF buffering and Extreme Learning Machine (ELM) is also being proposed to prevent work hunger in an SJF-MMBF system. Furthermore, the queues must be planned, which is usually used as a preventive fixed priority schedule. The results of the simulation show that the SJF-ELM is ideal inside strong duty as well as maximum is environment dynamically, with an efficient average employment hosting rate. Keywords: Delay-optimal virtual machine, scheduling algorithm, Shortest-Job-First, Min-Min Best Fit, Multi-level queue scheduling, VM-hosting durations and Particle SwarmOptimization.

show abstract

“…In [8], the authors built upon the HEFT algorithm presented in [41] to more explicitly consider the effect scheduling certain tasks before others will have on the performance of the DAG workflow. Biswas et al [7] extended the HEFT algorithm with additional heuristics and a multi-queue scheduler to address the problem of scheduling across heterogeneous systems. Al Ebrahim and Ahmad [1] also extended HEFT to explicitly consider all dependencies and data transfer among a static set of tasks.…”

Section: Related Workmentioning

confidence: 99%

SHADHO: Massively Scalable Hardware-Aware Distributed Hyperparameter Optimization

Kinnison

Kremer-Herman

Thain

et al. 2018

2018 IEEE Winter Conference on Applications of Computer Vision (WACV)

View full text Add to dashboard Cite

Computer vision is experiencing an AI renaissance, in which machine learning models are expediting important breakthroughs in academic research and commercial applications. Effectively training these models, however, is not trivial due in part to hyperparameters: user-configured values that control a model's ability to learn from data. Existing hyperparameter optimization methods are highly parallel but make no effort to balance the search across heterogeneous hardware or to prioritize searching high-impact spaces. In this paper, we introduce a framework for massively Scalable Hardware-Aware Distributed Hyperparameter Optimization (SHADHO). Our framework calculates the relative complexity of each search space and monitors performance on the learning task over all trials. These metrics are then used as heuristics to assign hyperparameters to distributed workers based on their hardware. We first demonstrate that our framework achieves double the throughput of a standard distributed hyperparameter optimization framework by optimizing SVM for MNIST using 150 distributed workers. We then conduct model search with SHADHO over the course of one week using 74 GPUs across two compute clusters to optimize U-Net for a cell segmentation task, discovering 515 models that achieve a lower validation loss than standard U-Net.

show abstract

Multi-level queue for task scheduling in heterogeneous distributed computing system

Cited by 16 publications

References 12 publications

Survey of Distributed Computing Frameworks for Supporting Big Data Analysis

Survey of Distributed Computing Frameworks for Supporting Big Data Analysis

SHADHO: Massively Scalable Hardware-Aware Distributed Hyperparameter Optimization

Contact Info

Product

Resources

About