MRemu: An Emulation-Based Framework for Datacenter Network Experimentation Using Realistic MapReduce Traffic

Abstract

As data volumes and the need for timely analysis grow, Big Data analytics frameworks have to scale out to hundred or even thousands of commodity servers. While such a scale-out is crucial to sustain desired computational throughput/latency and storage capacity, it comes at the cost of increased network traffic volumes and multiplicity of traffic patterns. Despite the sheer reality of the dependency between datacenter network (DCN) and time-to-insight through big data analysis, our experience as active networking researchers conveys that a large fraction of DCN research experimentation is conducted on network traces and/or synthetic flow traces. And while the respective results are often valuable as standalone contributions, in practice it turns out extremely difficult to quantitatively assess how the reported network optimization results translate to performance or faulttolerance improvement for actual analytics runtimes, e.g., due to the ability of these runtimes to overlap communication with computation. This paper presents MRemu, an emulation-based framework for conducting reproducible datacenter network research using accurate MapReduce workloads and at system scales that are relevant to the size of target deployments, albeit without requiring access to a hardware infrastructure of such scale. We choose the MapReduce (MR) framework as a design point, for it is a common representative of the most widely deployed frameworks for analysis of large volumes of -structured and unstructured -data and is reported to be highly sensitive to network performance. With MRemu, it is possible to quantify the impact of various network design parameters and softwaredefined control techniques to key performance indicators of a given MR application. We show through targeted experimental validation that MRemu exhibits high fidelity, when compared to the performance of MR applications on a real scale-out cluster of 16 high-end servers.

BigDataSDNSim: A simulator for analyzing big data applications in software‐defined cloud data centers

“…MaxiNet is capable of emulating large‐scale SDN cloud environments along with evaluating new SDN‐powered routing algorithms. Moreover, to allow Mininet to mimic the behaviors of MapReduce applications, MRemu 32 was introduced where it is capable of using realistic MapReduce workloads/traces within Mininet environments. It operates on latency periods extracted from MapReduce job traces (duration of tasks, waiting times, and so forth).…”

Uma Plataforma de Rede Definida por Software para Ambientes Convergentes de Computação Paralela

“…Para a experimentação, utilizamos a ferramenta MRemu [Neves et al 2015], um framework baseado em emulação para pesquisa em rede usando cargas de trabalho MapReduce. MRemu usa traços de rede, criados a partir de execuções de aplicações Hadoop em cluster reais, como base para produzir cargas de tráfego emuladas no Mininet.…”

BigDataSDNSim: A Simulator for Analyzing Big Data Applications in Software-Defined Cloud Data Centers

“…The main limitation of Mininet is that it does not support application-level infrastructures. To allow Mininet to mimic the behaviors of MapReduce applications, MRemu 16 was introduced where it is capable of using realistic MapReduce workloads/traces within Mininet environments. It operates based on latency periods extracted from MapReduce job traces (duration of tasks, waiting times, etc.).…”