A Constraint Programming Based Hadoop Scheduler for Handling MapReduce Jobs with Deadlines on Clouds

Abstract

A novel MapReduce constraint programming based matchmaking and scheduling algorithm (MRCP) that can handle MapReduce jobs with deadlines and achieve high system performance is devised. The MRCP algorithm is incorporated into Hadoop, which is a widely used open source implementation of the MapReduce programming model, as a new scheduler called the CP-Scheduler. This paper originates from the collaborative research with our industrial partner concerning the engineering of resource management middleware for high performance. It describes our experiences and the challenges that we encountered in designing and implementing the prototype CP-based Hadoop scheduler. A detailed performance evaluation of the CP-Scheduler is conducted on Amazon EC2 to determine the CP-Scheduler's effectiveness as well as to obtain insights into system behaviour and performance. In addition, the CP-Scheduler's performance is also compared with an earliest deadline first (EDF) Hadoop scheduler, which is implemented by extending Hadoop's default FIFO scheduler. The experimental results demonstrate the effectiveness of the CPScheduler's ability to handle an open stream of MapReduce jobs with deadlines in a Hadoop cluster.

PAS: Performance-Aware Job Scheduling for Big Data Processing Systems

“…Khan et al [15] applied linear regression method to perform runtime estimation for Hadoop jobs. Lim et al [16] proposed CP-Scheduler to estimate task execution time and handle MapReduce jobs with deadlines. Shao et al [17] proposed an energy-aware greedy algorithm (EAGA) for fine-grained task placement to minimize the energy consumption and job execution time.…”

“…Require: J kRequire: J k : the job set waiting for scheduling at τ k ; J c : a group of candidate scheduling sets; R k used : the used resource at τ k ; S fast ←arg mix j i ∈J r (t i .remain);(16 for j i ∈ J r do (17 t i .remain←t i .remain − t fast .remain; ALGORITHM 4: Continued.…”

Hadoop job scheduling with dynamic task splitting

“…LsPS [22] Balanced Pool Scheduler [18] Deadlines PD Scheduler [17] ARIA Scheduler [7] MRCP Scheduler [25] MIMP Scheduler [26] Table 1 Schedulers reviewed in this chapter.…”

“…Based on (24), the amount of time where a task has to wait for its slot to process is crucial to the performance of DTSS at different levels of splitting. When the waiting time for the data-local ( ) node is too small, the splitting point (P) will depends on the speed difference and the overheads incurred between processing on a non-data-local node and on a data-local node (25). If the N<<M, then P will tend to 0.…”

“…In contrast, under DTSS, it can be seen that cluster resources are shared equally among the users. For example, looking at the25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 105 109 113 117 121 125 129 133 137 141 145 149 DTSS, it can be seen that all users are granted an almost equal share of the cluster resources from epoch 11 to epoch 96. The cluster resources are then shared equally between user0 and user1 from epoch 97 to epoch 141 as user2 do not have any request for resources after epoch 96.…”