Scheduling MapReduce Jobs and Data Shuffle on Unrelated Processors

Abstract

We propose constant approximation algorithms for generalizations of the Flexible Flow Shop (FFS) problem which form a realistic model for non-preemptive scheduling in MapReduce systems. Our results concern the minimization of the total weighted completion time of a set of MapReduce jobs on unrelated processors and improve substantially on the model proposed by Moseley et al. (SPAA 2011) in two directions. First, we consider each job consisting of multiple Map and Reduce tasks, as this is the key idea behind MapReduce computations, and we propose a constant approximation algorithm. Then, we introduce into our model the crucial cost of data shuffle phase, i.e., the cost for the transmission of intermediate data from Map to Reduce tasks. In fact, we model this phase by an additional set of Shuffle tasks for each job and we manage to keep the same approximation ratio when they are scheduled on the same processors with the corresponding Reduce tasks and to provide also a constant ratio when they are scheduled on different processors. This is the most general setting of the FFS problem (with a special third stage) for which a constant approximation ratio is known.The authors were partially supported by the European Social Fund and Greek national resources under Thales-DELUGE and Heracleitus II programs. A short extended abstract of this work, including partial results, appeared in EDBT/ICDT 2014 Workshop on Algorithms for MapReduce and Beyond.

Scheduling MapReduce Jobs Under Multi-round Precedences

“…For single-round MapReduce scheduling, we obtain an 8-approximation algorithm. Our algorithm improves on the 12-approximation algorithm of [18,8], while refines their idea (of merging independent schedules of only map and only reduce tasks on their corresponding sets of processors into a single schedule) by applying a 2-approximation algorithm similar to that in [5, Lemma 6.1]. Note that [5] considers a set of orders of jobs, instead of jobs consisting of tasks, and the completion time of each order is specified by the completion of the job that finishes last.…”

“…First, in terms of modeling the MapReduce scheduling process: (i) We consider the practical scenario of multiround multi-task MapReduce jobs and capture their task dependencies, and (ii) we study both identical and unrelated processors, thus dealing with data locality. Second, in terms of algorithm design and analysis: (i) We propose an algorithmic framework for the multi-round MapReduce scheduling problem with proven performance guarantees, distinguishing between the case of indistinguishable and disjoint (map and reduce) sets of identical or unrelated processors, and (ii) our algorithms are based on natural LP relaxations of the problem and improve on the approximation ratios achieved in previous work [18,8]. Third, in terms of experimental analysis, we focus on the most general case of unrelated processors and show that our algorithms have an excellent performance in practice.…”

“…Our technique improves on ideas proposed in [8], by formulating an interval-indexed LP relaxation of the multi-round MapReduce scheduling problem so as to handle the multi-round precedences. Unlike previous work [18,8] we avoid the two-steps idea of creating partial schedules of only map and only reduce tasks and then combine them into one. As a result of this refinement, in the single-round case we show a 37.87-approximation for the single-round MapReduce scheduling problem, substantially improving on the previously proposed 54-approximation algorithm in [8].…”

“…Unlike previous work [18,8] we avoid the two-steps idea of creating partial schedules of only map and only reduce tasks and then combine them into one. As a result of this refinement, in the single-round case we show a 37.87-approximation for the single-round MapReduce scheduling problem, substantially improving on the previously proposed 54-approximation algorithm in [8]. Furthermore, we comment on the hardness of the multi-round MapReduce scheduling problem.…”

“…In Section 5, we compare our algorithms via simulations of random instances with a fast heuristic, proposed in [8], as well as with a lower bound on the optimal value of the multi-round MapReduce scheduling problem. To capture data locality issues, we consider instances that use processor and task relations.…”