Configuration optimization method of Hadoop system performance based on genetic simulated annealing algorithm

Luo, Xin; Fu, Xueliang

doi:10.1007/s10586-018-2029-y

Cited by 11 publications

(11 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As the first step of genetic algorithm design, the selection of coding method must fully consider the needs of practical problems and the influence of coding method on the design of genetic operators (especially crossover and mutation operators) [ 15 ].…”

Section: Dynamic Road Network Model and Dynamic Optimal Path Modelmentioning

confidence: 99%

Multimedia Urban Road Path Optimization Based on Genetic Algorithm

Fang-fang

2022

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

In order to study multimedia urban road path optimization based on genetic algorithm, a dynamic path optimization based on genetic algorithm is proposed. Firstly, for the current situation of traffic congestion, time constraints are strictly considered based on the traditional hard time window logistics distribution vehicle scheduling problem model. Then, the mathematical model is established, and the optimal solution is solved by the combination of decomposition coordination algorithm and genetic algorithm. We divide multiple customers into different customer groups and determine the service object order of each express car in each customer group, so as to obtain the most valuable scheduling scheme. Finally, in the process of solving the model, the relevant and reliable distribution basis for enterprise distribution is collected, including customer geographical coordinates, demand, delivery time window, unit cost required for loading and unloading, loading and unloading time, and penalty cost to be borne by distribution enterprises after early arrival and late arrival. Using the improved genetic algorithm, the optimal solution of each objective function is actually obtained in about 140 generations, which is faster than that before the improvement. Using the genetic algorithm based on sequence coding, a hybrid genetic algorithm is constructed to solve the model problem. Through the comparative analysis of experimental data, it is known that the algorithm has good performance, is a feasible algorithm to solve the VSP problem with time window, and can quickly obtain the vehicle routing scheduling scheme with reference value.

show abstract

Section: Dynamic Road Network Model and Dynamic Optimal Path Modelmentioning

confidence: 99%

Multimedia Urban Road Path Optimization Based on Genetic Algorithm

Fang-fang

2022

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

show abstract

“…Several studies focused on improving Hadoop performance [8][9][10]. Aydin et al [11] developed a Hadoop program for distributed data log analysis with a small cloud.…”

Section: Literature Reviewmentioning

confidence: 99%

“…Based on the memory requirements, a metaheuristic algorithm called a genetic simulated annealing algorithm was proposed to adjust the major configuration parameters of Hadoop. The drawback of this optimization method is the high processing time, which is largely due to the convergence of the initial guess [9]. A heterogeneous job allocation scheduler with a dynamic grouping integrated neighboring search algorithm was proposed in [27].…”

Section: Problem Statementmentioning

confidence: 99%

See 1 more Smart Citation

A Multi-Optimization Technique for Improvement of Hadoop Performance with a Dynamic Job Execution Method Based on Artificial Neural Network

et al. 2020

View full text Add to dashboard Cite

The improvement of Hadoop performance has received considerable attention from researchers in cloud computing fields. Most studies have focused on improving the performance of a Hadoop cluster. Notably, various parameters are required to configure Hadoop and must be adjusted to improve performance. This paper proposes a mechanism to improve Hadoop, schedule jobs, and allocate and utilize resources. Specifically, we present an improved ant colony optimization method to schedule jobs according to the job size and the time expected for execution. Priority is given to the job with the minimum data size and minimum response time. The resource usage and running jobs by data node are predicted using an artificial neural network, and job activity and resource usage are monitored using the resource manager. Moreover, we enhance the Hadoop Name node performance by adding an aggregator node to the default HDFS framework architecture. The changes involve four entities: the name node, secondary name node, aggregator nodes, and data nodes, where the aggregator node is responsible for assigning the jobs among the data node, and the Name node keeps tracking only the aggregator nodes. We test the overall scheme among Amazon EC2 and S3, and show the results of throughput and CPU response time for different data sizes. Finally, we show that the proposed approach shows significant improvement compare to native Hadoop and other approaches.

show abstract

“…Aside from scaling up/down the cluster, to achieve a certain performance level, Hadoop can also be optimized at the application level by managing the file placement [3], or by managing the jobs [4]. In this paper, the authors improve Hadoop performance by optimizing the job concurrency.…”

Section: Introductionmentioning

confidence: 99%

Asynchronous Non-Blocking Algorithm to Handle Straggler Reduce Tasks in Hadoop System

Khoiruddin¹,

Zakaria²,

Alhussian³

2020

Int. J. Adv. Sci. Eng. Inf. Technol.

View full text Add to dashboard Cite

Hadoop is widely adopted as a big data processing application as it can run on commercial hardware at a reasonable time. Hadoop uses asynchronous blocking concurrency using Thread and Future class. Therefore, in some cases such as network link or hardware failure, a running task may block other tasks from running (the task becomes straggler). Hadoop releases are equipped with algorithms to handle straggler tasks problem. However, the algorithms manage Map and Reduce task similarly, while the straggler root cause might be different for both tasks. In this paper, the Asynchronous Non-Blocking (ANB) method is proposed to improve the performance and avoid the blocking of Reduce task in Hadoop. Instead of using the single queue, our approach uses two queues, i.e. task queue and callback queue. When a task is not ready or detected as a straggler, it is removed from the main task queue and temporarily sent to the callback queue. When the task is ready to run, it will be sent back to the main task queue for running. The performance of the algorithm is compared with rTuner, the latest paper found on handling straggler task in Reduce task. From the comparison, it is shown that ANB consistently gives faster time to complete because any unready tasks will be directly put into the callback queue without blocking other tasks. Furthermore, the overhead time in rTuner is high as it needs to check the straggler status and to find the reason for a task to become straggler.

show abstract

Configuration optimization method of Hadoop system performance based on genetic simulated annealing algorithm

Cited by 11 publications

References 7 publications

Multimedia Urban Road Path Optimization Based on Genetic Algorithm

Multimedia Urban Road Path Optimization Based on Genetic Algorithm

A Multi-Optimization Technique for Improvement of Hadoop Performance with a Dynamic Job Execution Method Based on Artificial Neural Network

Asynchronous Non-Blocking Algorithm to Handle Straggler Reduce Tasks in Hadoop System

Contact Info

Product

Resources

About