Using Hadoop Technology to Overcome Big Data Problems by Choosing Proposed Cost-efficient Scheduler Algorithm for Heterogeneous Hadoop System (BD3)

Hussein, Abou_el_ela Abdou

doi:10.9734/jsrr/2020/v26i930310

Cited by 11 publications

(2 citation statements)

References 18 publications

(21 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Because the disk space of each node is different in a heterogeneous cluster environment, the theoretical amount of space used for each node calculated may be greater than the maximum load of the node. Each private cluster has independent schedulers and uses its own scheduling algorithm to manipulate its own resources, reducing the competitive requirements of the operations on resources [27]. In addition, the jobs run independently in their own clusters, and jobs from different users can be executed simultaneously.…”

Section: Methodsmentioning

confidence: 99%

Load Balancing Algorithms for Hadoop Cluster in Unbalanced Environment

Wang

2022

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

Considering that in the process of job scheduling, the cluster load should be prebalanced rather than remedied when the load is seriously unbalanced; therefore, in this paper, the task scheduling flow of the Hadoop cluster is analyzed deeply. On the Hadoop platform, a self-dividing algorithm is proposed for load balancing. An intelligent optimization algorithm is used to solve load balance. A dynamic feedback load balancing scheduling method is proposed from the point of view of task scheduling. In order to solve the shortcoming of the fair scheduling algorithm, this paper proposes two ways to improve the resource utilization and overall performance of Hadoop. When the mapping task is completed and the tasks to be reduced are assigned, the task assignment is based on the performance of the nodes to be reduced. It gives full play to the advantages of the ant colony algorithm and the hive colony algorithm so that the fusion algorithm can better deal with load balance. Then, three existing scheduling algorithms are introduced in detail: single queue scheduling, capacity scheduling, and fair scheduling. On this basis, an improved task scheduling strategy based on genetic algorithm is proposed to allocate and execute application tasks to reduce task completion time. The experiment verifies the effectiveness of the algorithm. The LBNP algorithm greatly improves the efficiency of reducing task execution and job execution. The delay capacity scheduling algorithm can ensure that most tasks can achieve localization scheduling, improve resource utilization, improve load balance, and speed up job completion time.

show abstract

Section: Methodsmentioning

confidence: 99%

Load Balancing Algorithms for Hadoop Cluster in Unbalanced Environment

Wang

2022

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

show abstract

“…When used with care, simple procedures can be beneficial [57] VANTAGE: Provision of composite systems…”

Section: Vane: In the Right Directionmentioning

confidence: 99%

Domain and Challenges of Big Data and Archaeological Photogrammetry With Blockchain

2022

View full text Add to dashboard Cite

With gigantic growth of data volume that is moved across the web links today, there has been a gigantic measure of perplexing information produced. Extremely huge sets of data including universities, organizations framework, institution gas, petroleum sector, healthcare, that have so enormous thus complex information with more differed structure. The major challenge is how to handle this significant volume of data, also in archaeological photogrammetry which alluded to as Big Data. Although big data has to be securely flying and conveyed through the internet. It cannot be controlled with regular conventional methods that fail to handle it, so there is a need for more up-to-date developed tools. The big data have frequently divided into V's characteristics beginning from three V's: volume, velocity and variety. The initial three V's have been stretched out during time through researches to arrive 56 V's till now. Among them are three newfound by the author that implies it multiplied near twenty times. Researcher had to dive to search for all of these characteristics in many researches to detect and build comparisons to answer the old, current, and restored essential inquiry, "how many V's aspects (characteristics) in big data with archaeological photogrammetry and blockchain". This paper provides a comprehensive overview of all secured big data V's (characteristics) as well as their strength and limitations with archaeological photogrammetry and blockchain.

show abstract

Cloud Resource Hadoop Cluster Scheduling Algorithm Based on Evolutionary Artificial Bee Colony Model for Mobile Sustainable Networks

Fan¹

2022

Evolutionary Computing and Mobile Sustainable Networks

View full text Add to dashboard Cite

Using Hadoop Technology to Overcome Big Data Problems by Choosing Proposed Cost-efficient Scheduler Algorithm for Heterogeneous Hadoop System (BD3)

Cited by 11 publications

References 18 publications

Load Balancing Algorithms for Hadoop Cluster in Unbalanced Environment

Load Balancing Algorithms for Hadoop Cluster in Unbalanced Environment

Domain and Challenges of Big Data and Archaeological Photogrammetry With Blockchain

Cloud Resource Hadoop Cluster Scheduling Algorithm Based on Evolutionary Artificial Bee Colony Model for Mobile Sustainable Networks

Contact Info

Product

Resources

About