BIGhybrid -- A Toolkit for Simulating MapReduce in Hybrid Infrastructures

Abstract

Abstract-Cloud computing has increasingly been used as a platform for running large business and data processing applications. Although clouds have become highly popular, when it comes to data processing, the cost of usage is not negligible. Conversely, Desktop Grids, have been used by a plethora of projects, taking advantage of the high number of resources provided for free by volunteers. Merging cloud computing and desktop grids into hybrid infrastructure can provide a feasible low-cost solution for big data analysis. Although frameworks like MapReduce have been conceived to exploit commodity hardware, their use on hybrid infrastructure poses some challenges due to large resource heterogeneity and high churn rate. This study introduces BIGhybrid a toolkit to simulate MapReduce on hybrid environments. The main goal is to provide a framework for developers and system designers to address the issues of hybrid MapReduce. In this paper, we describe the framework which simulates the assembly of two existing middleware: BitDewMapReduce for Desktop Grids and Hadoop-BlobSeer for Cloud Computing. Experimental results included in this work demonstrate the feasibility of our approach.

SMART: An Application Framework for Real Time Big Data Analysis on Heterogeneous Cloud Environments

“…Grid'5000 is an experimental testbed carried out under the Inria ALADDIN development plan with support from CNRS, RENATER and several universities in France. The simulation is performed with the BIGhybrid simulator, introduced in previous work [3], with a focus on cloud and hybrid systems 3 .…”

“…If one considers MR, two distinct DFS implementations may be required to handle data distribution in two scenarios, namely low-bandwidth and high-bandwidth. Building on previous work performed on MR for hybrid environments [19], [3], Figure 1 illustrates the solution proposed here to model a hybrid system which depicts a Global Dispatcher and Global Aggregator to be used on the infrastructure for services that use multiple data abstractions. The Global Dispatcher located outside the cloud has middleware functions for handling task assignment, and management of user-provided data.…”

“…It is a centralized data storage system that manages policies for splitting data and distributing it in accordance with the needs of each system. The working principle is similar to a publish/subscribe service in which the system obtains data and publishes computing results [3]. The Global Aggregator obtains data output from both systems and merges them in order to obtain the final data set.…”

“…Specifically related to this concern, data streaming is an important technique to support incoming data with fast input as continual streams generate large volumes of data. It is not unusual to use resources from multiple data centers and, increasingly more common, from several clouds for deploying clusters for Big Data analysis [3].…”

BIGhybrid: a simulator for MapReduce applications in hybrid distributed infrastructures validated with the Grid5000 experimental platform

“…BIGhybrid is a toolkit for MR simulation in hybrid environments and was previously introduced in , with a focus on Cloud and DG. The simulator itself is based on the SimGrid framework .…”