The Case for Evaluating MapReduce Performance Using Workload Suites

Abstract

Abstract-MapReduce systems face enormous challenges due to increasing growth, diversity, and consolidation of the data and computation involved. Provisioning, configuring, and managing large-scale MapReduce clusters require realistic, workloadspecific performance insights that existing MapReduce benchmarks are ill-equipped to supply.In this paper, we build the case for going beyond benchmarks for MapReduce performance evaluations. We analyze and compare two production MapReduce traces to develop a vocabulary for describing MapReduce workloads. We show that existing benchmarks fail to capture rich workload characteristics observed in traces, and propose a framework to synthesize and execute representative workloads. We demonstrate that performance evaluations using realistic workloads gives cluster operator new ways to identify workload-specific resource bottlenecks, and workload-specific choice of MapReduce task schedulers.We expect that once available, workload suites would allow cluster operators to accomplish previously challenging tasks beyond what we can now imagine, thus serving as a useful tool to help design and manage MapReduce systems.

Workload Modeling for Computer Systems Performance Evaluation

“…Chen et al describe MapReduce workloads from 6 months on a 600-machine cluster and 1.5 months on a 3000-machine cluster at Facebook, from 3 weeks on a cluster at Yahoo!, and from several other installations [125,124]. Some of their data is shown in Figures 9.34 and 9.35.…”

“…For example, consider the data about MapReduce workloads at Facebook available from the SWIM project [125]. MapReduce applications have two stages, a map stage and a reduce stage (this is explained in the box on page 498).…”

“…Chen at al. have claimed that due to the complex nature of the MapReduce workloads and their many attributes, which typically do not conform to well-known distributions, using randomly mixed samples of traced data is better than using benchmarks or models [125].…”

“…Each file is the concatenation of 24 random one-hour samples from the original log file [125,124]. Available from GitHub: URL https://github.com/SWIMProjectUCB/SWIM/wiki.…”

Benchmarking in the Cloud: What It Should, Can, and Cannot Be

“…To compare the resulting service they need a benchmark build around a set of representative analytical tasks. Most research in the area in done on actual MapReduce benchmarks like MRBench [28] or designing appropriate MapReduce workloads [29]. Pavlo et al [30] show how to have analytical workload run by both MapReduce and Distributed Databases and compare the results.…”