Load balancing in MapReduce environments for data intensive applications

Jing

Concurrency and Computation

et al. 2016

Self Cite

Summary At present MapReduce computing model‐based Hadoop framework has gradually become the most famous distributed computing framework because of its remarkable features such as scalability, fault tolerance, data security, and powerful IO ability. However, Hadoop framework only supports limited load balancing policies, which may result in performance deterioration in heterogeneous clusters. Additionally Hadoop does not have advanced dynamic load balancing mechanism in enabling its optimal performance in dynamic environment. This paper presents a sliding window‐based dynamic load balancing algorithm, which specially aims at balancing the load among the heterogeneous nodes during the Hadoop job processing. The presented algorithm is evaluated in both simulated and physical environments. The experimental results show that the performances in terms of efficiency of Hadoop cluster can be significantly improved. Copyright © 2016 John Wiley & Sons, Ltd.

Section: Simulationsmentioning

confidence: 99%

A sliding window‐based dynamic load balancing for heterogeneous Hadoop clusters

Jing

Concurrency and Computation

et al. 2016

Self Cite

“…MapReduce has been taken up by the community in dealing with data‐intensive applications . However, MapReduce only offers simple job‐scheduling schemes which may deteriorate the performance significantly in heterogeneous computing environments . Fan et al .…”

Section: Related Workmentioning

confidence: 99%

“…MapReduce has been taken up by the community in dealing with data-intensive applications [23][24][25]. However, MapReduce only offers simple job-scheduling schemes which may deteriorate the performance significantly in heterogeneous computing environments [24,[26][27][28]. Fan et al [29] proposed a reducer-phase-based load-balancing algorithm; however, mappers are highly time-consuming as computational tasks are usually executed by mappers.…”

Section: Introductionmentioning

confidence: 99%

A MapReduce‐based parallel K‐means clustering for large‐scale CIM data verification

Deng

Concurrency and Computation

et al. 2015

Self Cite

The Common Information Model (CIM) has been heavily used in electric power grids for data exchange among a number of auxiliary systems such as communication systems, monitoring systems and marketing systems. With an rapid deployment of digitalized devices in electric power networks, the volume of data continuously grows which makes verification of CIM data a challenging issue. This paper presents a parallel K-means for large scale CIM data verification based on the MapReduce computing model which has been widely taken up by the community in dealing with data intensive applications. By distributing the CIM data into a number of computers in a MapReduce cluster environment, the computation in CIM data verification is significantly improved. Furthermore, a load balancing scheme is designed to balance the workloads among the heterogeneous MapReduce computing nodes for a further improvement in computation efficiency. The performance of the parallel K-means clustering in CIM data verification is first evaluated in a small scale experimental MapReduce cluster and subsequently evaluated in a large scale simulation environment.

“…After getting the support of each item, we construct distributed FPtree to mine frequent itemsets. Inappropriate task allocation in a distributed environment will reduce the utilization of resources, seriously affect the efficiency of program execution [10] [11]. If the task is assigned unreasonably, it will cause the following: the execution time of the computer A is T, in a wait state because the task is finished; the execution time of the computer B is 10T.…”

Section: • Weighing the Allocation Of Tasksmentioning

confidence: 99%

Research of the FP-Growth Algorithm Based on Cloud Environments

Zhou¹,

Wang²

2014

JSW

The emergence of cloud computing solves the problems that traditional data mining algorithms encounter when dealing with large data. This paper studies the FP-Growth algorithm and proposes a parallel linked list-based FPG algorithm based on MapReduce programming model, named as the PLFPG algorithm. And then it describes the main idea of algorithm. Finally, by using different data sets to test the algorithm, the experimental result shows that PLFPG algorithm has higher efficiency and better flexibility and scalability.