A Review of Scheduling Algorithms in Hadoop

Sharma, Añil; Singh, Gurdeep

doi:10.1007/978-3-030-29407-6_11

Cited by 8 publications

(6 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The fair scheduler, although suitable for single-tenant environments, faces limitations in performance based on dynamic job grouping and submission at runtime. Sharma and Singh [7] delved into big data, Hadoop ecosystem tools, and scheduling algorithms for the MapReduce model and conducted a comparative analysis. They examined the strengths and weaknesses of schedulers in various Hadoop deployments for heterogeneous use cases.…”

Section: Significance Of Studymentioning

confidence: 99%

Turbocharging Hadoop Fair Scheduler using Dynamic Job Grouping in Multi-Job Workloads

Dhamee,

Chitrakar

2024

Preprint

View full text Add to dashboard Cite

show abstract

Section: Significance Of Studymentioning

confidence: 99%

Turbocharging Hadoop Fair Scheduler using Dynamic Job Grouping in Multi-Job Workloads

Dhamee,

Chitrakar

2024

Preprint

View full text Add to dashboard Cite

show abstract

“…FIFO is the original and the default Hadoop scheduler. The main goal is to schedule jobs based on the job arrival in the queue (First-in First-out) [11], [12]. Resources will be allocated to the job in the frontmost of the queue.…”

Section: B Hadoop Schedulermentioning

confidence: 99%

“…Fair Scheduler is a method for assigning resources to jobs such that all jobs get a nearly equal share of resources [11], [12]. Fair scheduler organises jobs employing resource pools and fairly shares resources between these pools.…”

Section: B Hadoop Schedulermentioning

confidence: 99%

See 1 more Smart Citation

Asynchronous Non-Blocking Algorithm to Handle Straggler Reduce Tasks in Hadoop System

Khoiruddin¹,

Zakaria²,

Alhussian³

2020

Int. J. Adv. Sci. Eng. Inf. Technol.

View full text Add to dashboard Cite

Hadoop is widely adopted as a big data processing application as it can run on commercial hardware at a reasonable time. Hadoop uses asynchronous blocking concurrency using Thread and Future class. Therefore, in some cases such as network link or hardware failure, a running task may block other tasks from running (the task becomes straggler). Hadoop releases are equipped with algorithms to handle straggler tasks problem. However, the algorithms manage Map and Reduce task similarly, while the straggler root cause might be different for both tasks. In this paper, the Asynchronous Non-Blocking (ANB) method is proposed to improve the performance and avoid the blocking of Reduce task in Hadoop. Instead of using the single queue, our approach uses two queues, i.e. task queue and callback queue. When a task is not ready or detected as a straggler, it is removed from the main task queue and temporarily sent to the callback queue. When the task is ready to run, it will be sent back to the main task queue for running. The performance of the algorithm is compared with rTuner, the latest paper found on handling straggler task in Reduce task. From the comparison, it is shown that ANB consistently gives faster time to complete because any unready tasks will be directly put into the callback queue without blocking other tasks. Furthermore, the overhead time in rTuner is high as it needs to check the straggler status and to find the reason for a task to become straggler.

show abstract

“…Furthermore, a job scheduling policy gives a dominant impact on the performance of distributed computing systems [11]. Several scheduling policies were introduced to balance workload efficiently and minimize the waiting time of jobs in distributed systems [12]- [14]. However, few researchers, have managed efficient job scheduling algorithms under deadline constraints in cloud-based environments.…”

Section: Introductionmentioning

confidence: 99%

LSTPD: Least Slack Time-Based Preemptive Deadline Constraint Scheduler for Hadoop Clusters

et al. 2020

View full text Add to dashboard Cite

Big data refers to numerous forms of complex and large datasets which need distinctive computational platforms in order to be analyzed. Hadoop is one of the popular frameworks for analytics of big data. In Hadoop, a big job is split into multiple small tasks and then they are distributed to worker nodes in a parallel way using MapReduce to speed up computational processes. In this aspect, it is important how to improve throughput performance. MapReduce jobs require quick responses from the worker nodes to complete them under their deadlines. The existing scheduling schemes for Hadoop such as FIFO, fair, and capacity schedulers cannot guarantee the quick response requirement satisfying a prior deadline. Thus, Hadoop system needs to improve response time and completion time for the heterogeneous MapReduce jobs. In this paper, we propose an efficient preemptive deadline constraint scheduler based on least slack time and data locality. In order for better allocation of tasks and load balancing, we first analyze the task scheduling behaviors of the Hadoop platform. Based on that, we propose a novel preemptive approach which considers the remaining execution time of the job being executed in deciding preemption. The experimental results show that the proposed scheme significantly reduces the job execution time and queue waiting time, compared to existing schemes.

show abstract

A Review of Scheduling Algorithms in Hadoop

Cited by 8 publications

References 28 publications

Turbocharging Hadoop Fair Scheduler using Dynamic Job Grouping in Multi-Job Workloads

Turbocharging Hadoop Fair Scheduler using Dynamic Job Grouping in Multi-Job Workloads

Asynchronous Non-Blocking Algorithm to Handle Straggler Reduce Tasks in Hadoop System

LSTPD: Least Slack Time-Based Preemptive Deadline Constraint Scheduler for Hadoop Clusters

Contact Info

Product

Resources

About