Center-of-Gravity Reduce Task Scheduling to Lower MapReduce Network Traffic

Abstract

MapReduce is by far one of the most successful realizations of large-scale data-intensive cloud computing platforms. MapReduce automatically parallelizes computation by running multiple map and/or reduce tasks over distributed data across multiple machines.Hadoop is an open source implementation of MapReduce. When Hadoop schedules reduce tasks, it neither exploits data locality nor addresses partitioning skew present in some MapReduce applications. This might lead to increased cluster network traffic. In this paper we investigate the problems of data locality and partitioning skew in Hadoop. We propose Center-of-Gravity Reduce Scheduler (CoGRS), a locality-aware skew-aware reduce task scheduler for saving MapReduce network traffic. In an attempt to exploit data locality, CoGRS schedules each reduce task at its center-ofgravity node, which is computed after considering partitioning skew as well. We implemented CoGRS in Hadoop-0.20.2 and tested it on a private cloud as well as on Amazon EC2. As compared to native Hadoop, our results show that CoGRS minimizes off-rack network traffic by averages of 9.6% and 38.6% on our private cloud and on an Amazon EC2 cluster, respectively. This reflects on job execution times and provides an improvement of up to 23.8%.

Reconfigurable network testbed for evaluation of datacenter topologies

“…Our work is the first step in expanding on some of their ideas. Other research has focused on the reduce task scheduling [14,15] and the partitioning function [19]. As these works look to schedule tasks and partitions to improve shuffle traffic performance, we look to schedule the network to optimize application performance.…”

A self-adaptive scheduling algorithm for reduce start time

“…As the MapReduce distributed computations were analyzed as a divisible load scheduling problem [21], several classes of algorithms were proposed and examined for scheduling divisible loads on a heterogeneous system with memory limits [22]. Some task scheduling algorithms are to release the data communication among remote slots, e.g., the center-of-gravity reduce scheduler is a localityaware and skew-aware reduce task scheduler for saving MapReduce network traffic [23], and MaRCO employs eager reduce to process partial data from some map tasks while overlapping with other map tasks' communication [6].…”

The Ambient Scrutinize of Scheduling Algorithms in Big Data Territory.

“…The M. Hammoud, M. Rehman, et al developed Center-of-Gravity Reduce Scheduling Algorithm [45]. This technique to designs a locality-aware, skew-aware reduce task scheduler for stop MapReduce network traffic.…”