Research on Scheduling Scheme for Hadoop Clusters

Xie, Jiong; Meng, Fanjun; Wang, Hailong; Pan, Hongfang; Cheng, JinHong; Qin, Xiao

doi:10.1016/j.procs.2013.05.423

Cited by 20 publications

(9 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most of these schedulers suffer from the same problems as discussed for all of the above mentioned schedulers. For example, Xie et al (2013) proposed a new scheduler that estimates execution time and prefetching input data before assigning new tasks for computing nodes. This algorithm proposes a new shuffle schema to improve system performance.…”

Section: Background and Related Workmentioning

confidence: 99%

A scalable Map Reduce tasks scheduling: a threading-based approach

Althebyan

AlQudah

Jararweh

et al. 2017

IJCSE

View full text Add to dashboard Cite

Abstract:The Map Reduce paradigm is now considered a standard platform that is used for large-scale data processing and management. A major operation that the Map Reduce platform relies on greatly is tasks scheduling. Although many schedulers have been presented, task scheduling is still one of the major problems that face Map Reduce frameworks. Schedulers need to maintain data locality to achieve an acceptable performance by avoiding several data transmissions. Hence, in this paper, we propose a new scheduling algorithm named 'MTL' that utilises multi-threading principles. The MTL scheduler assigns a dedicated thread for each data block. Indeed, the multi-threading approach shows great results that make our MTL scheduler a scalable one that performs well. At the same time, it maintains the locality property. During the evaluation of the MTL scheduler performance, two main factors were taken into consideration; the simulation time and the energy consumption. The MTL scheduler is then compared with other existing schedulers such as FIFO, matchmaking, and delay schedulers. The MTL scheduler showed favourable results and proved its advantages over other existing schedulers.

show abstract

Section: Background and Related Workmentioning

confidence: 99%

A scalable Map Reduce tasks scheduling: a threading-based approach

Althebyan

AlQudah

Jararweh

et al. 2017

IJCSE

View full text Add to dashboard Cite

show abstract

“…So it improves the performance lower than our method. A predictive scheduler and prefetching mechanism [21] are proposed to improve the performance of MapReduce by assigning two tasks to each slot. Unfortunately, it affects the priority of jobs.…”

Section: Related Workmentioning

confidence: 99%

Scheduling algorithm based on prefetching in MapReduce clusters

Sun

Zhuang

et al. 2016

Applied Soft Computing

View full text Add to dashboard Cite

Due to cluster resource competition and task scheduling policy, some map tasks are assigned to nodes without input data, which causes significant data access delay. Data locality is becoming one of the most critical factors to affect performance of MapReduce clusters. As machines in MapReduce clusters have large memory capacities, which are often underutilized, in-memory prefetching input data is an effective way to improve data locality. However, it is still posing serious challenges to cluster designers on what and when to prefetch. To effectively use prefetching, we have built HPSO (High Performance Scheduling Optimizer), a prefetching service based task scheduler to improve data locality for MapReduce jobs. The basic idea is to predict the most appropriate nodes for future map tasks based on current pending tasks and then preload the needed data to memory without any delaying on launching new tasks. To this end, we have implemented HPSO in Hadoop-1.1.2. The experiment results have shown that the method can reduce the map tasks causing remote data delay, and improves the performance of Hadoop clusters.

show abstract

“…As the most widely adopted software stack for dataintensive computing [9]- [11], especially for cloud computing [12]- [14], Hadoop [15] was primarily inspired by concepts developed by the Google File System [16] and MapReduce [17]. Because of its prominence, increasingly number of studies have focused on improving the performance of Hadoop, and most of which are related to either task scheduling or job scheduling [18]- [23].…”

Section: Related Workmentioning

confidence: 99%

“…Vanderwiel et al [24] studied data prefetching mechanisms and proposed several data prefetching strategies to eliminate the effect of low degree data locality. Xie et al [11] presented a policy to prefetch data after predicting the next task to be scheduled, and discussed when, what, and how much to be prefetched. Seo et al [9] applied prefetching and preshuffling strategy to optimize the overall performance of a shared Hadoop environment and reduced the execution time of MapReduce by up to 73%.…”

Section: Related Workmentioning

confidence: 99%

BOLAS: Bipartite-Graph Oriented Locality-Aware Scheduling for MapReduce Tasks

Xue

Gao

et al. 2015

2015 14th International Symposium on Parallel and Distributed Computing

View full text Add to dashboard Cite

Task scheduling is critical to reduce the makespan of MapReduce jobs. It is an effective approach for scheduling optimization by improving the data locality, which involves attempting to locate a task and its related data block on the same node. However, recent studies have been insufficient in addressing the locality issue. This paper proposes BOLAS, a MapReduce task scheduling algorithm, which models the scheduling process as a bipartite-graph matching problem trying best to assign data block to the nearest task. By considering the divergence of node performance of distribution of data blocks in MapReduce cluster, BOLAS can achieve a high degree of data locality, guarantee minimal data transfer during execution, and reduces a job's makespan subsequently. As a dynamic algorithm, BOLAS solves the model using Kuhn-Munkres optimal matching algorithm, and can be deployed in either homogeneous or heterogeneous environments. In this study, BOLAS was implemented as a plugin for Hadoop, and the experimental results indicate that BOLAS can localize nearly 100% of the map tasks and reduce the execution time by up to 67.1%.

show abstract

Research on Scheduling Scheme for Hadoop Clusters

Cited by 20 publications

References 4 publications

A scalable Map Reduce tasks scheduling: a threading-based approach

A scalable Map Reduce tasks scheduling: a threading-based approach

Scheduling algorithm based on prefetching in MapReduce clusters

BOLAS: Bipartite-Graph Oriented Locality-Aware Scheduling for MapReduce Tasks

Contact Info

Product

Resources

About