A Multi-Optimization Technique for Improvement of Hadoop Performance with a Dynamic Job Execution Method Based on Artificial Neural Network

Rayan, Alanazi; Alhazmi, Fawaz; Chung, Haejin; Nah, Yunmook

doi:10.1007/s42979-020-00182-3

Cited by 12 publications

(6 citation statements)

References 27 publications

(33 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This method takes into account both task size and expected execution time. In addition, they have improved Hadoop performance by integrating an aggregation node into the default architecture of the HDFS distributed file system [11]. Other research has exploited AEML, an acceleration engine specially designed to balance the workload across multiple GPUs.…”

Section: Related Work On Load Balancingmentioning

confidence: 99%

Load Balancing Large Data Sets in a Hadoop Cluster

Anselme A,

Alain N,

Francis

et al. 2023

IJDPS

View full text Add to dashboard Cite

With the interconnection of one to many computers online, the data shared by users is multiplying daily. As a result, the amount of data to be processed by dedicated servers rises very quickly. However, the instantaneous increase in the volume of data to be processed by the server comes up against latency during processing. This requires a model to manage the distribution of tasks across several machines. This article presents a study of load balancing for large data sets on a cluster of Hadoop nodes. In this paper, we use Mapreduce to implement parallel programming and Yarn to monitor task execution and submission in a node cluster.

show abstract

Section: Related Work On Load Balancingmentioning

confidence: 99%

Load Balancing Large Data Sets in a Hadoop Cluster

Anselme A,

Alain N,

Francis

et al. 2023

IJDPS

View full text Add to dashboard Cite

show abstract

“…Based on the energy consumption in each node, a decision is made to launch map and/or reduce tasks. An ACO algorithm is used in [22] to finalize the job execution in a batch based on heterogeneous job size and its expected latency. However, the job taking less data and response time is given high priority in the job schedule.…”

Section: Literature Surveymentioning

confidence: 99%

Optimizing MapReduce Task Scheduling on Virtualized Heterogeneous Environments Using Ant Colony Optimization

Jeyaraj

Paul

2022

IEEE Access

View full text Add to dashboard Cite

Consuming Hadoop MapReduce via virtual infrastructure as a service is becoming common practice as cloud service providers (CSP) offers relevant applications and scalable resources. One of the predominant requirements of cloud users is to improve resource utilization in the virtual cluster during the service period. However, it may not be possible when MapReduce workloads and virtual machines (VM) are highly heterogeneous. Therefore, in this paper, we addressed these heterogeneities and proposed an efficient MapReduce scheduler to improve resource utilization by placing the right combination of the map and reduce tasks in each VM in the virtual cluster. To achieve this, we transformed the MapReduce task scheduling problem into a 2-Dimensional (2D) bin packing model and obtained an optimal schedule using the ant colony optimization (ACO) algorithm. As an added advantage, our proposed ACO based bin packing (ACO-BP) scheduler minimized the makespan for a batch of jobs. To showcase the performance improvement, we compared our proposed scheduler with three existing schedulers that work well in a heterogeneous environment. As expected, results show that ACO-BP significantly outperformed the existing schedulers while dealing with workload and VM level heterogeneities.

show abstract

“…The proposed schedulers were not evaluated in either dynamic or heterogeneous environments. Alanazi et al [20] proposed to give priority to the jobs with the minimum data size and response time for job scheduling. To achieve that, they proposed an artificial neural network to predict resource usage and running jobs by Hadoop data nodes.…”

Section: Related Workmentioning

confidence: 99%

The Performance Optimization of Big Data Processing by Adaptive MapReduce Workflow

Tang

2022

IEEE Access

View full text Add to dashboard Cite

The discussion context of this paper is big data processing of MapReduce by volunteer computing in dynamic and opportunistic environments. This paper conducts a series of simulations to explore the relationship between the overall performance of volunteer overlays responding to different workload of big data problems. The discovery from the simulations includes some optimization points in overlay size, going over which by adding more volunteers brings little benefit for the overall performance. Based on the discovery of optimization points, this paper proposes a bootstrapping protocol, which can adapt volunteers into variable-sizes overlays, enabling workflow of single-round MapReduce or multipleround MapReduce, and a single or multiple overlays for each round. The variable overlays aim to create adaptive workflow during MapReduce processing, so that the optimization points can be caught. As another benefit, the unnecessary computing-capacities can be released during computing when the optimization points are reached. The case study shows a few optimization workflows that are formed by the proposed bootstrapping protocol to process the big data cases. The workflows lead to the optimization points and dynamically balance the workload at the same time. The experiment results have demonstrated that the optimization strategies have either achieved 36% or 71% higher performance than the plain MapReduce workflow and minimized the use of computing resources by releasing 12.5% to 75% volunteers during computing, where the original plain MapReduce must hold all the volunteers to the end of computing. The extensibility of the simulation parameterization to more diverse real-world applications have been clarified.

show abstract

A Multi-Optimization Technique for Improvement of Hadoop Performance with a Dynamic Job Execution Method Based on Artificial Neural Network

Cited by 12 publications

References 27 publications

Load Balancing Large Data Sets in a Hadoop Cluster

Load Balancing Large Data Sets in a Hadoop Cluster

Optimizing MapReduce Task Scheduling on Virtualized Heterogeneous Environments Using Ant Colony Optimization

The Performance Optimization of Big Data Processing by Adaptive MapReduce Workflow

Contact Info

Product

Resources

About