Performance-driven task co-scheduling for MapReduce environments

Polo, Jordà; Carrera, David; Becerra, Yolanda; Steinder, Małgorzata; Whalley, Ian

doi:10.1109/noms.2010.5488494

Cited by 144 publications

(111 citation statements)

References 4 publications

Supporting

Mentioning

111

Contrasting

Order By: Relevance

“…Polo et al [21] proposed a performancedriven task co-scheduling algorithm, which estimates the required finish time for each task and allocates resources prior for the tasks that cannot be completed timely. Kc and Anyanwu [22] proposed a deadline constraint (DC) scheduler, which tries to allocate a fixed number of Map jobs to each task according to the size of tasks and assumes that each task can utilize all job slots in the Reduce step.…”

Section: Related Workmentioning

confidence: 99%

Profit-oriented task scheduling algorithm in Hadoop cluster

Chai

Dong

2016

J Embedded Systems

View full text Add to dashboard Cite

Nowadays, many enterprises provide cloud services based on their own Hadoop clusters. Because the resources of a Hadoop cluster are limited, the Hadoop cluster must select some specific tasks to allocate limited resources in order to get the maximal profit. In this paper, we study the maximal profit problem for a given candidate task set. We describe the candidate task set with a valid sequence and propose a sequence-based scheduling strategy. In order to improve the efficiency of finding a valid sequence, we design some pruning strategies and give the corresponding scheduling algorithm. Finally, we propose a timeout handling algorithm when some task runs timeout. Experiments show that the total profit of the proposed algorithm is very close to the ideal maxima and is obviously bigger than related scheduling algorithms under different experimental settings.

show abstract

Section: Related Workmentioning

confidence: 99%

Profit-oriented task scheduling algorithm in Hadoop cluster

Chai

Dong

2016

J Embedded Systems

View full text Add to dashboard Cite

show abstract

“…The former is used to decide task placement on TaskTrackers over time, and is the main object of study of this paper. The latter is used to estimate the number of tasks to be run in parallel for each job in order to meet some performance objectives, expressed in RAS in the form of completion time goals, and was extensively evaluated and validated in [6].…”

Section: Resource-aware Adaptive Schedulermentioning

confidence: 99%

“…Notice that the scenario we focus on is very dynamic, with jobs entering and leaving the system unpredictably, so the goal of this component is to provide estimates of s j req that guide resource allocation. This component leverages the techniques already described in [6] and therefore we will not provide further details in this paper.…”

Section: Fig 1 System Architecturementioning

confidence: 99%

“…One of the seminal works in this space is [6]. In this work, the authors propose a scheduling scheme that enables soft-deadline support for MapReduce jobs.…”

Section: Related Workmentioning

confidence: 99%

“…This loose coupling between scheduling and resource management limits the opportunity to efficiently control the utilization of resources in the system. Providing support for user-specified 'goals' in MapReduce clusters is also challenging, due to high variability induced by the presence of outlier tasks (tasks that take much longer than other tasks) [5][6][7][8]. Solutions to mitigate the detrimental impact of such outliers typically rely on scheduling techniques such as speculative scheduling [9], and killing and restarting of tasks [5].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Resource-Aware Adaptive Scheduling for MapReduce Clusters

Polo

Castillo²,

Carrera

et al. 2011

Lecture Notes in Computer Science

Self Cite

111

View full text Add to dashboard Cite

Abstract. We present a resource-aware scheduling technique for MapReduce multi-job workloads that aims at improving resource utilization across machines while observing completion time goals. Existing MapReduce schedulers define a static number of slots to represent the capacity of a cluster, creating a fixed number of execution slots per machine. This abstraction works for homogeneous workloads, but fails to capture the different resource requirements of individual jobs in multiuser environments. Our technique leverages job profiling information to dynamically adjust the number of slots on each machine, as well as workload placement across them, to maximize the resource utilization of the cluster. In addition, our technique is guided by user-provided completion time goals for each job. Source code of our prototype is available at [1].

show abstract

MapReduce delay scheduling with deadline constraint

Wei

et al. 2013

Concurrency and Computation

View full text Add to dashboard Cite

SUMMARYMapReduce programming paradigm has been widely applied to solve large-scale data-intensive problems. Intensive studies of MapReduce scheduling have been carried out to improve MapReduce system performance. Delay scheduling is a common way to achieve high data locality and system performance. However, inappropriate delays can lead to low system throughput and potentially break the original job priority constraints. This paper proposes a deadline-enabled delay (DLD) scheduling algorithm that optimizes job delay decisions according to real-time resource availability and resource competition, while still meets job deadline constraints. Experimental results illustrate that the resource availability estimation method of DLD is accurate (92%). Compared with other approaches, DLD reduces job turnaround time by 22% in average while keeping a high locality rate (88%).

show abstract

Performance-driven task co-scheduling for MapReduce environments

Cited by 144 publications

References 4 publications

Profit-oriented task scheduling algorithm in Hadoop cluster

Profit-oriented task scheduling algorithm in Hadoop cluster

Resource-Aware Adaptive Scheduling for MapReduce Clusters

MapReduce delay scheduling with deadline constraint

Contact Info

Product

Resources

About