New Scheduling Algorithms for Improving Performance and Resource Utilization in Hadoop YARN Clusters

“…Hadoop core consists of a Hadoop distributed file system storage and a MapReduce processing [22]. This article is based on Yet Another Resource Negotiator (YARN) framework introduced by Yahoo in 2010 [23]. As in Fig.…”

Section: Introductionmentioning

confidence: 99%

RETRACTED ARTICLE: Detecting straggler MapReduce tasks in big data processing infrastructure by neural network

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Initially, non-local executions for respective blocks and frequent failed tasks are traced from the logs to identify where repeated executions happening in multi-cloud heterogeneous environment. To minimize makespan and improve resource utilization for a batch of MapReduce jobs in heterogeneous environment, a novel task scheduler is proposed in [9]. It includes two policies HaSTE, HaSTE-A for YARN distributed system.…”

Section: Literature Surveymentioning

confidence: 99%

Handling Non-Local Executions to Improve MapReduce Performance Using Ant Colony Optimization

et al. 2021

View full text Add to dashboard Cite

Improving the performance of the MapReduce scheduler is a primary objective, especially in a heterogeneous virtual cloud environment. A map task is assigned with an input split(IS) which consists of one or more data blocks. When a map task is assigned to more than one data block, non-local execution is performed. In classical MapReduce scheduling schemes, data blocks are copied over the network to a node in where the map task is running. This increases job latency and consumes more network bandwidth within and between racks in the cloud data-center. Considering this situation, we propose a methodology "improving data locality using ant colony optimization (IDLACO)" to minimize the number of non-local executions and virtual network bandwidth consumption when IS are assigned to more than one data block. First IDLACO determines a list of an optimal number of data blocks for each map task of a job to perform a non-local execution reducing the job latency and virtual network consumption. Then, the target virtual machine to execute the map task is determined on the basis of its heterogeneous performance. Finally, if a set of data blocks is transferred to the same node for repeated job execution, it is decided to temporarily cache those data block in the target virtual machine. The performance of IDLACO is analysed and compared with fair scheduler and Holistic scheduler based on the parameters, such as the number of non-local executions, average map task latency, job latency, and amount of bandwidth consumed for a MapReduce job. Results show that our proposed IDLACO significantly outperforms the classical fair scheduler and Holistic scheduler.

show abstract

“…A large number of studies [18][19][20][21][22][23][27][28][29][30] have been conducted to minimize the makespan of jobs and improve Hadoop performance. We classified the works into two categories: (i) Studies ignoring resource and workload heterogeneity, (ii) Studies considering the heterogeneity in terms of resource and workload.…”

Section: Related Workmentioning

confidence: 99%

“…With the premise of improving Hadoop performance in terms of makespan, Yao et al [27] have presented a new scheduler for a batch of MapReduce jobs. The proposed schedulers use the information of requested resources, resource capacities and dependency between tasks which constitutes the tasks' fitness for scheduling.…”

Section: Related Workmentioning

confidence: 99%

TMaR: a two-stage MapReduce scheduler for heterogeneous environments

Maleki

Faragardi

Rahmani

et al. 2020

Hum. Cent. Comput. Inf. Sci.

View full text Add to dashboard Cite

In the context of MapReduce task scheduling, many algorithms mainly focus on the scheduling of Reduce tasks with the assumption that scheduling of Map tasks is already done. However, in the cloud deployments of MapReduce, the input data is located on remote storage which indicates the importance of the scheduling of Map tasks as well. In this paper, we propose a two-stage Map and Reduce task scheduler for heterogeneous environments, called TMaR. TMaR schedules Map and Reduce tasks on the servers that minimize the task finish time in each stage, respectively. We employ a dynamic partition binder for Reduce tasks in the Reduce stage to lighten the shuffling traffic. Indeed, TMaR minimizes the makespan of a batch of tasks in heterogeneous environments while considering the network traffic. The simulation results demonstrate that TMaR outperforms Hadoop-stock and Hadoop-A in terms of makespan and network traffic and achieves by an average of 29%, 36%, and 14% performance using Wordcount, Sort, and Grep benchmarks. Besides, the power reduction of TMaR is up to 12%.

show abstract

New Scheduling Algorithms for Improving Performance and Resource Utilization in Hadoop YARN Clusters

Cited by 34 publications

References 24 publications

RETRACTED ARTICLE: Detecting straggler MapReduce tasks in big data processing infrastructure by neural network

RETRACTED ARTICLE: Detecting straggler MapReduce tasks in big data processing infrastructure by neural network

Handling Non-Local Executions to Improve MapReduce Performance Using Ant Colony Optimization

TMaR: a two-stage MapReduce scheduler for heterogeneous environments

Contact Info

Product

Resources

About