An Optimal Task Selection Scheme for Hadoop Scheduling

Suresh, S.; Gopalan, N. P.

doi:10.1016/j.ieri.2014.09.093

Cited by 17 publications

(12 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In a study on optimal task, selection scheme for Hadoop scheduling is made, in which when the multiple jobs are submitted decision on selecting a task is made by the fair scheduler. In this paper, more importance is on improving the performance by helping the scheduler when multiple local tasks are available for a node [1,11]. In the next paper, the effectiveness of two main key factors like data locality and data skew is examined on homogeneous and heterogeneous clusters.…”

Section: Related Studymentioning

confidence: 99%

“…It handles jobs by handling them expeditiously by playacting task splits parallelly and assigns it to the nodes of the network. The major challenge within the distributed systems is the way to assign the tasks to the nodes and to handle the nodes of the network during a fair manner; specified nodes are network of distributed systems square measure equally loaded [11].…”

Section: Distributed Systemsmentioning

confidence: 99%

See 1 more Smart Citation

Optimized Approach (SPCA) for Load Balancing in Distributed HDFS Cluster

Manjula¹,

Sundaram²

2020

SN COMPUT. SCI.

View full text Add to dashboard Cite

Hadoop is an open-source utility which allows users to provide massive input in terms of data and facilitates the computation. Role of Hadoop in load balancing is enormous which allows the user to configure the network of nodes having master/slave nodes. Hadoop's typical architecture takes into consideration the default configuration for the machine as homogeneous, but many of the real-time application or clusters of nodes will have the homogeneous configurations. Thus, an effort is made in this paper to consider the homogeneity of the nodes in clusters and build an efficient algorithm which does load balancing in an efficient way when compared with the default balancer of Hadoop which works well only on homogeneous nodes.

show abstract

Section: Related Studymentioning

confidence: 99%

Section: Distributed Systemsmentioning

confidence: 99%

Optimized Approach (SPCA) for Load Balancing in Distributed HDFS Cluster

Manjula¹,

Sundaram²

2020

SN COMPUT. SCI.

View full text Add to dashboard Cite

show abstract

“…Data locality is also considered in the scheduling of big data computing systems. To improve the performance, authors in [19] propose an optimal task selection algorithm for better data locality and fairness. In addition, job characteristics are taken into consideration in the job-aware scheduling algorithms.…”

Section: Related Workmentioning

confidence: 99%

“…On the other hand, if P 1 cannot be satisfied at time t + 1 in SD, and, LD has redundant resources, it enlarges δ (line 9-11). If both P 1 and P 2 cannot be met by estimated resources, we sort the jobs in each category by their resource demands r i , and start from the job with smallest resource demand, try to assign as many jobs as possible to utilize resources (line [12][13][14][15][16][17][18][19][20]. After the assignments, each of the categories may have some leftover (A c1 and A c2 may larger than 0).…”

Section: Dynamic Configuration For Reserved Resource Ratiomentioning

confidence: 99%

DRESS: Dynamic RESource-Reservation Scheme for Congested Data-Intensive Computing Platforms

Mao

Green

Wang

et al. 2018

2018 IEEE 11th International Conference on Cloud Computing (CLOUD)

View full text Add to dashboard Cite

In the past few years, we have envisioned an increasing number of businesses start driving by big data analytics, such as Amazon recommendations and Google Advertisements. At the back-end side, the businesses are powered by big data processing platforms to quickly extract information and make decisions. Running on top of a computing cluster, those platforms utilize scheduling algorithms to allocate resources. An efficient scheduler is crucial to the system performance due to limited resources, e.g. CPU and Memory, and a large number of user demands. However, besides requests from clients and current status of the system, it has limited knowledge about execution length of the running jobs, and incoming jobs' resource demands, which make assigning resources a challenging task. If most of the resources are occupied by a long-running job, other jobs will have to keep waiting until it releases them. This paper presents a new scheduling strategy, named DRESS that particularly aims to optimize the allocation among jobs with various demands. Specifically, it classifies the jobs into two categories based on their requests, reserves a portion of resources for each of category, and dynamically adjusts the reserved ratio by monitoring the pending requests and estimating release patterns of running jobs. The results demonstrate DRESS significantly reduces the completion time for one category, up to 76.1% in our experiments, and in the meanwhile, maintains a stable overall system performance.

show abstract

“…Job scheduling is an additional task of resource management [4], [10], [11], [12], [13], which is one of the most indispensable components of cluster-level infrastructure layers. YARN [8] is a distributed resource management system for resource allocation in compute clusters [2], [4] and job scheduling, in the particular case of Hadoop MapReduce.…”

Section: Introductionmentioning

confidence: 99%

Building block components to control a data rate in the Apache Hadoop compute platform

et al. 2015

2015 18th International Conference on Intelligence in Next Generation Networks

View full text Add to dashboard Cite

Abstract-Resource management is one of the most indispensable components of cluster-level infrastructure layers. Users of such systems should be able to specify their job requirements as a configuration parameter (CPU, RAM, disk I/O, network I/O) and have the scheduler translate those into an appropriate reservation and allocation of resources. YARN is an emerging resource management in the Hadoop ecosystem, which supports only RAM and CPU reservation at present.In this paper, we propose a solution that takes into account the operation of the Hadoop Distributed File System to control the data rate of applications in the framework of a Hadoop compute platform. We utilize the property that a data pipe between a container and a DataNode consists of a disk I/O subpipe and a TCP/IP subpipe. We have implemented building block software components to control the data rate of data pipes between containers and DataNodes and provide a proof-of-concept with measurement results.

show abstract

An Optimal Task Selection Scheme for Hadoop Scheduling

Cited by 17 publications

References 4 publications

Optimized Approach (SPCA) for Load Balancing in Distributed HDFS Cluster

Optimized Approach (SPCA) for Load Balancing in Distributed HDFS Cluster

DRESS: Dynamic RESource-Reservation Scheme for Congested Data-Intensive Computing Platforms

Building block components to control a data rate in the Apache Hadoop compute platform

Contact Info

Product

Resources

About