On scheduling in map-reduce and flow-shops

Abstract

The map-reduce paradigm is now standard in industry and academia for processing large-scale data. In this work, we formalize job scheduling in map-reduce as a novel generalization of the two-stage classical flexible flow shop (FFS) problem: instead of a single task at each stage, a job now consists of a set of tasks per stage. For this generalization, we consider the problem of minimizing the total flowtime and give an efficient 12-approximation in the offline setting and an online (1 + Motivated by map-reduce, we revisit the two-stage flow shop problem, where we give a dynamic program for minimizing the total flowtime when all jobs arrive at the same time. If there are fixed number of job-types the dynamic program yields a PTAS; it is also a QPTAS when the processing times of jobs are polynomially bounded. This gives the first improvement in approximation of flowtime for the two-stage flow shop problem since the trivial 2-approximation algorithm of Gonzalez and Sahni [29] in 1978, and the first known approximation for the FFS problem. We then consider the generalization of the two-stage FFS problem to the unrelated machines case, where we give an offline 6-approximation and an online (1 + )-speed O( 1 4 )-competitive algorithm.

Optimization of Slot and Map Reduce Workload

“…Rather than rely on hardware to deliver highavailability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures. The problem of map-reduceScheduling [3] by abstracting the above requirements and desiderata inscheduling terms. In particular, we focus on multiple-task multiple-machine two-stage non-migratory scheduling with precedence constraints; these constraints exist between each map task and reducetask for a job.We consider a subset of [4]production workload that consists of MapReduce jobs with nodependencies.…”

Joint optimization of overlapping phases in MapReduce

“…Note that the particular form of the phase overlapping in MapReduce makes data processing different from the traditional multiple stage processing models in manufacturing and communication networks [16,24,38]. In particular, in a typical tandem queueing model [38] (or flowshop model [16,24]) a job must complete processing at one station before moving to the following station. In contrast, in MapReduce a job may have tasks that start the shuffle phase before all the tasks complete the map phase.…”

“…To provide context for this result, it is useful to consider the related literature from the flowshop scheduling community, e.g., [10,16,19,24]. In particular, the flowshop literature considers traditional tandem queues (which do not allow overlapping) with all jobs arrive at time zero and tend to focus on a different performance metric: makespan.…”

An Efficiency-Aware Scheduling for Data-Intensive Computations on MapReduce Clusters

“…Hammoud et al propose center-of-gravity reduce task scheduling aiming to lower MapReduce network traffic [25], which model reduce input distribution as mass distribution model, by properly assign reduce tasks to save network cost, so we can call it data locality in reduce phase, which is not a consideration in our study but in future work. Benjamin Moseley et al study the scheduling problem in MapReduce and Flowshops [26], which formalize job scheduling in map-reduce as a novel generalization of the two-stage classical flexible flow shop (FFS) problem: instead of a single task at each stage, a job now consists of a set of tasks per stage. He et al develop a new MapReduce scheduling technique to enhance map task's data locality named as MaBtchmaking [27], which is a very good and efficient scheduling technique, motivated by improving data locality of map tasks on the basis of well known in-used Hadoop schedulers, such as FIFO and Hadoop Fair Scheduler.…”