A sliding window‐based dynamic load balancing for heterogeneous Hadoop clusters

Liu, Yang; Jing, Weizhe; Liu, Youbo; Liu, Lin; Qi, Man; Xiang, Yang

doi:10.1002/cpe.3763

Cited by 9 publications

(9 citation statements)

References 23 publications

(35 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The aforementioned researches indicate that the parallelized BPNN benefits from the advantages of the Hadoop framework. However, several researches pointed out that the load imbalance issue of the distributed training significantly impacts the data processing efficiency in a heterogeneous Hadoop cluster [17], [19], [29]. Although Hadoop has a number of default schedulers such as FIFO, fair scheduler, and capacity scheduler, these universal schedulers cannot adapt to various kinds of Hadoop jobs [14]- [16].…”

Section: Introductionmentioning

confidence: 99%

“…Although Hadoop has a number of default schedulers such as FIFO, fair scheduler, and capacity scheduler, these universal schedulers cannot adapt to various kinds of Hadoop jobs [14]- [16]. Additionally scheduler designed for a specific type of Hadoop job [14]- [16] may not serve the other types of the jobs well [17]. Research [17] also presented a load balancing algorithm which can serve multiple types of Hadoop jobs.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Resource Aware Parallelized Back Propagation Neural Network in Enabling Efficient Large-Scale Digital Health Data Processing

Liu

Chen

et al. 2019

IEEE Access

Self Cite

View full text Add to dashboard Cite

Along with the development of digital health, efficient machine learning is anxiously needed to handle the growing health data. Among various machine learning algorithms, back propagation neural network (BPNN) shows great effectiveness in both academia and industrial fields. However, it is frequently reported that the conventional BPNN algorithm encounters low efficiency issue in dealing with large-scale digital health data. Therefore this paper presents a Hadoop based parallelized BPNN algorithm which is able to process the large-scale data efficiently. In order to complement the potential accuracy loss issue for the parallelized data processing, ensemble learning techniques are also involved. Additionally although Hadoop supplies a number of default schedulers, the heterogeneous distributed computing environment may still impact the efficiency of the parallelized BPNN. Consequently, this paper also presents a gene expression programming (GEP) algorithm based load balancing approach, which enables the computing resource awareness and the optimal scheduling of the parallelized BPNN. The experiments employ the classification task as the underlying testing basis. Two types of the experiments are carried out, in which the first one focuses on evaluating the accuracy of the presented algorithm with classifying the benchmark dataset; the second one focuses on evaluating the efficiency of the presented algorithm with classifying the large-scale dataset. The experimental results show the effectiveness of the presented resource aware parallelized BPNN algorithm. INDEX TERMS Back propagation neural network, parallelization, Hadoop, load balancing, gene expression programming.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

A Resource Aware Parallelized Back Propagation Neural Network in Enabling Efficient Large-Scale Digital Health Data Processing

Liu

Chen

et al. 2019

IEEE Access

Self Cite

View full text Add to dashboard Cite

show abstract

“…Duan [1] used the CPU utilization rate, disk utilization ratio, page error number, request number, request response time and other relevant indicators to calculate the real-time load of the server. Gang [2] proposed a method to classify the user requested services to allocate the system resources, so as to achieve dynamic load balancing. Shailesh [3] used a fuzzy dynamic load balancing algorithm to achieve load balancing through task scheduling.…”

Section: Related Workmentioning

confidence: 99%

“…Therefore, it is important that special attention should be given to restructure the system behaviors in accordance with the changes in user behaviors by balancing the system load. Existing system load balancing methods [1][2][3] are mainly based on resource allocation and task scheduling strategies, but not consider when and how to dynamically reconstruct the system behaviors for the real-time load equilibrium.…”

Section: Introductionmentioning

confidence: 99%

Behavior Reconstruction Models for Large-scale Network Service Systems

Zhang

Liu

Wang

et al. 2018

Peer-to-Peer Netw. Appl.

View full text Add to dashboard Cite

In large-scale network service systems, the phenomenon of instantaneous gathering of a large number of users can cause system abnormality, whenever the load imposed by the user behaviors does not match the system load. This paper proposes a behavior reconstruction model for large-scale network service systems integrated with Petri net reconstruction methodology, for the purpose of achieving load balancing in the system under increasing number of users. Based on the features of the user interaction behavior sequence, the behavioral load balancing model defines a user behavior membership function. Then, a random fuzzy Petri net with delay is presented to control the user behavior reconstruction. Experiments conducted by considering various changes in the number of user behaviors and their distribution in unit time demonstrate that the proposed methodology can effectively trigger the reconstructed model to balance the system load when the system load exceeds the defined warning point.

show abstract

“…However, Hadoop does not have a sophisticated scheme in job scheduling. For this purpose, Liu et al present a dynamic load balancing algorithm on the basis of sliding windows with an aim to target at heterogeneous Hadoop cluster systems. Hadoop has over 190 parameters for user to configure.…”

mentioning

confidence: 99%

Internet of People

2016

Concurrency and Computation

View full text Add to dashboard Cite

Internet of People (IoP) refers to digital connectivity of people through the Internet infrastructure forming a network of collective intelligence and stimulating interactive communication among people. The purpose of the special issue is to collate a selection of representative articles that were primarily presented at the IEEE International Conference on Internet of People (IoP 2015) on August 10 to 14, 2015, in Beijing, China. The special issue was also made open to public submissions for a wide inclusion.The scope of the special issue is broad and represents a multidisciplinary nature of IoP. It covers topics from the network enabling technologies at the physical layer to services at the application layer. It is reported that around 40% of the world population has an Internet connection today, and the number of Internet users reached 3 billion in 2014. The sheer size of the Internet population has led to big data challenges of volume, velocity, variety, and veracity of the digital data generated. For this purpose, this special issue also includes a few articles on high performance computing techniques that can speed up the computation process in analyzing big volumes of data.Mobile and wireless networks facilitate the communication among people. In the work of Huang et al.[1] they present a multiple-user partner selection algorithm to improve the overall secrecy rate of the network. Zhong et al. Crowdsourcing enables IoP by soliciting contributions from a large group of people connected by the Internet. Wang et al.[5] conduct a survey on mobile crowdsourcing from the aspects of realtime and location-sensitive crowdsourced tasks. Related research challenges and possible solutions are discussed. Shao et al.[6] use crowdsensing in vehicular networks to predict the traffic condition in intelligent transportation systems. This solves the problem of traditional approaches being inefficiency ineffectiveness in data uploading and usage. Guo et al. [7] extract information from external-related attributes and improve latent Dirichlet allocation to build a topic mining model to facilitate services in telephone call centers. Knowledge diffusion as a component of crowdsourcing also plays an increasing role in modern communication networks and social networks. Zhang et al.[8] study the topological structure and research a new knowledge diffusion model taking into account both learning and forgetting attributes. The results from this research reveal that the social networks with a high degree of heterogeneity well suit for knowledge diffusion. In a mobile environment in IoP, network latency would have a significant impact on the communication of mobile services. For this purpose, Ding et al. [9] target at service composition so that mobile services can be optimized and provisioned to users with low communication latency. In the work of Gopalakrishna et al.[10] they assess relevance in cyber-physical systems. For this purpose, a new metric called relevance score is proposed for evaluation of a number of machine learning t...

show abstract

A sliding window‐based dynamic load balancing for heterogeneous Hadoop clusters

Cited by 9 publications

References 23 publications

A Resource Aware Parallelized Back Propagation Neural Network in Enabling Efficient Large-Scale Digital Health Data Processing

A Resource Aware Parallelized Back Propagation Neural Network in Enabling Efficient Large-Scale Digital Health Data Processing

Behavior Reconstruction Models for Large-scale Network Service Systems

Internet of People

Contact Info

Product

Resources

About