Scheduling algorithm based on prefetching in MapReduce clusters

Abstract

Due to cluster resource competition and task scheduling policy, some map tasks are assigned to nodes without input data, which causes significant data access delay. Data locality is becoming one of the most critical factors to affect performance of MapReduce clusters. As machines in MapReduce clusters have large memory capacities, which are often underutilized, in-memory prefetching input data is an effective way to improve data locality. However, it is still posing serious challenges to cluster designers on what and when to prefetch. To effectively use prefetching, we have built HPSO (High Performance Scheduling Optimizer), a prefetching service based task scheduler to improve data locality for MapReduce jobs. The basic idea is to predict the most appropriate nodes for future map tasks based on current pending tasks and then preload the needed data to memory without any delaying on launching new tasks. To this end, we have implemented HPSO in Hadoop-1.1.2. The experiment results have shown that the method can reduce the map tasks causing remote data delay, and improves the performance of Hadoop clusters.

RETRACTED ARTICLE: Detecting straggler MapReduce tasks in big data processing infrastructure by neural network

“…As data are transferred in the copy and shuffle stages, they have the most significant impact at execution time. There is an assigned weight for each stage, which is the ratio of the stage's execution time to the total execution time [28,29,32,33,34,35,36]. It is also possible to calculate errors by comparing the estimated weights by a real one [15].…”

Two-Stage Job Scheduling Model Based on Revenues and Resources

“…Job scheduling in the big data platform is crucial to the optimization of the platform performance. In order to improve the efficiency of the job execution and optimize the performance of the platform, the researches [8,9,10] propose data placement strategy and job scheduling algorithm based on the minimum data transmission time to reduce the data transmission time and improve the efficiency of the job execution. However, the revenue is not considered in the algorithms.…”

“…A set of jobs E j is found to be executed at T time and the required resources of E j do not conflict with pre-allocated resources (lines 2-8). If E j is not null, the job is selected that makes the waste resource rate minimize and the optimal start time of the job is T (lines [10][11][12][13][14][15][16]. If E j is null, T is set to the start time of the next time period in P R (line 18).…”

Empirical Study of Job Scheduling Algorithms in Hadoop MapReduce

“…A research work in [22] proposed HPSO (High-Performance Scheduling Optimizer) that enhances the scheduling algorithm based on pre-fetching of data needed to map tasks. This pre-fetching reduces the time needed to transfer data over the network and hence improves the performance.…”