Reconfigurable network testbed for evaluation of datacenter topologies

Moody, William Clay; Anderson, Jason; Wange, Kuang-Ching; Apon, Amy

doi:10.1145/2608020.2608023

Cited by 6 publications

(2 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some studies in the literature use MapReduce-like traffic patterns to evaluate their research [9], [10], [6]. They normally model the MR shuffle pattern as map tasks (typically one per node) sending data to one or more reduce tasks.…”

Section: Introductionmentioning

confidence: 99%

“…For example, it is possible to compare the performance (e.g., job completion time) of a given MapReduce job on different network topologies and network control software. The MapReduce emulation also works as a stand-alone tool that can be used to run network experiments in clusters with limited resources (i.e., not robust enough to run real Hadoop jobs [10]). MRemu sports following features and functionality:…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

MRemu: An Emulation-Based Framework for Datacenter Network Experimentation Using Realistic MapReduce Traffic

Neves

Rose

Katrinis

2015

2015 IEEE 23rd International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems

View full text Add to dashboard Cite

As data volumes and the need for timely analysis grow, Big Data analytics frameworks have to scale out to hundred or even thousands of commodity servers. While such a scale-out is crucial to sustain desired computational throughput/latency and storage capacity, it comes at the cost of increased network traffic volumes and multiplicity of traffic patterns. Despite the sheer reality of the dependency between datacenter network (DCN) and time-to-insight through big data analysis, our experience as active networking researchers conveys that a large fraction of DCN research experimentation is conducted on network traces and/or synthetic flow traces. And while the respective results are often valuable as standalone contributions, in practice it turns out extremely difficult to quantitatively assess how the reported network optimization results translate to performance or faulttolerance improvement for actual analytics runtimes, e.g., due to the ability of these runtimes to overlap communication with computation. This paper presents MRemu, an emulation-based framework for conducting reproducible datacenter network research using accurate MapReduce workloads and at system scales that are relevant to the size of target deployments, albeit without requiring access to a hardware infrastructure of such scale. We choose the MapReduce (MR) framework as a design point, for it is a common representative of the most widely deployed frameworks for analysis of large volumes of -structured and unstructured -data and is reported to be highly sensitive to network performance. With MRemu, it is possible to quantify the impact of various network design parameters and softwaredefined control techniques to key performance indicators of a given MR application. We show through targeted experimental validation that MRemu exhibits high fidelity, when compared to the performance of MR applications on a real scale-out cluster of 16 high-end servers.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%