Performance optimization of computing task scheduling based on the Hadoop big data platform

Abstract

Hadoop, a distributed computing framework that can efficiently process large-scale datasets, has been used by an increasing number of organizations as the basic computing framework to build cloud computing platforms. Improving its execution efficiency is a hot research direction in the industry, and the scheduling problem is a key factor affecting the execution efficiency of Hadoop. It is very important to identify its shortcomings and improve them. This paper examines and analyses the optimization of computing task scheduling performance based on the Hadoop big data platform. This paper first analyses Hadoop big data processing. Hadoop has high scalability. Computing nodes can be added at any time, and they can participate in cluster work through simple configuration. The paper discusses the improvement in the Hadoop resource scheduling algorithm. The task scheduling algorithm in the Hadoop-based data task localization proposed in this paper is compared with the default algorithm used in the Hadoop task scheduling algorithm. The former shows better local data in all four jobs, there are more data localization tasks, and the expected goal is achieved. The effectiveness of the algorithm is verified, and the performance is improved by 30%.

Integrated Data, Task and Resource Management to Speed Up Processing Small Files in Hadoop Cluster

“…The average execution time in the proposed solution for word count application is 11% lower compared to Sharma et al [29], 20.4% lower compared to Li et al [23] and 15.7% lower compared to Li et al [30]. The average execution time in the proposed solution for K-means clustering application is 8.73 lower compared to Sharma et al [29], 16% lower compared to Li et al [23] and 12% lower compared to Li et al [30]. The application speedup has increased in the proposed solution due to combined three factors high relevant task grouping, fine grained data locality and popularity based data block replication at a fine grained level.…”

“…4 and 5. The average execution time in the proposed solution for word count application is 11% lower compared to Sharma et al [29], 20.4% lower compared to Li et al [23] and 15.7% lower compared to Li et al [30]. The average execution time in the proposed solution for K-means clustering application is 8.73 lower compared to Sharma et al [29], 16% lower compared to Li et al [23] and 12% lower compared to Li et al [30].…”

“…Small text dataset of 100000,200000,300000 and 400000 files with file size from 1KB to 10 MB is used for experimentation. Three recent works in category of data management (Dynamic repository approach by Sharma et al [29]), task management (PAS scheme by Li et al [23]) and resource management (genetic algorithm based resource optimization scheme by Li et al [30]). Each of the above works attempted to speed up execution from three different perspectives of data, task and resource management and thus the proposed solution was compared with these works to prove the effectiveness of joint management of data, task and resources.…”

“…Li et al [30] Improved genetic algorithm is used to schedule the resources to the tasks with goal of minimizing the task completion time.…”

“…Due to this, the make span of Map reduce jobs increased in this work. Li et al [30] minimized execution time through optimal allocation of tasks to resources. But the work did not consider task collocation based on data sharing characteristics.…”