Scalable data management for map-reduce-based data-intensive applications: a view for cloud and hybrid infrastructures

Concurrency and Computation

Geyer

2015

Self Cite

Summary Cloud computing has increasingly been used as a platform for running large business and data processing applications. Conversely, Desktop Grids have been successfully employed in a wide range of projects, because they are able to take advantage of a large number of resources provided free of charge by volunteers. A hybrid infrastructure created from the combination of Cloud and Desktop Grids infrastructures can provide a low‐cost and scalable solution for Big Data analysis. Although frameworks like MapReduce have been designed to exploit commodity hardware, their ability to take advantage of a hybrid infrastructure poses significant challenges because of their large resource heterogeneity and high churn rate. In this paper, BIGhybrid is proposed, a simulator for two existing classes of MapReduce runtime environments: BitDew‐MapReduce designed for Desktop Grids and BlobSeer‐Hadoop designed for Cloud computing, where the goal is to carry out accurate simulations of MapReduce executions in a hybrid infrastructure composed of Cloud computing and Desktop Grid resources. This work describes the principles of the simulator and describes the validation of BIGhybrid with the Grid5000 experimental platform. Owing to BIGhybrid, developers can investigate and evaluate new algorithms to enable MapReduce to be executed in hybrid infrastructures. This includes topics such as resource allocation and data splitting. Copyright © 2015 John Wiley & Sons, Ltd.

Section: Background and Related Workmentioning

confidence: 99%

mentioning

confidence: 99%

BIGhybrid: a simulator for MapReduce applications in hybrid distributed infrastructures validated with the Grid5000 experimental platform

Anjos

Concurrency and Computation

Geyer

2015

Self Cite

“…Concerning the implementation of Active Data, we plan to investigate rollback mechanisms for fault-tolerant execution of applications and evaluate distributed implementations of the publish/subscribe substrate. Finally, several application prototypes are being developed using Active Data: a MapReduce runtime which mixes low power mobile devices (tablets, set-top boxes, smartphones) and online Cloud storage [31], a distributed and cooperative content delivery network to distribute virtual appliance images embedding large HEP applications to Internet Desktop Grid resources [53] and a distributed network of checkpoint image server featuring server selection using network distance [54].…”

Section: Resultsmentioning

confidence: 99%

“…For instance, [28][29][30] investigate several options to efficiently deliver data to Cloud, Grids and Desktop Grids infrastructures using P2P systems. More recently, several works have investigated the possibility to execute MapReduce applications on Clouds and Desktop Clouds [31,32]. Our preliminary work around Active Data [33,34] have explored issue of representing data sets when these are distributed on hybrid infrastructures.…”

Section: Related Workmentioning

confidence: 99%

Active Data: A programming model to manage data life cycle across heterogeneous systems and infrastructures

Simonet

Future Generation Computer Systems

Ripeanu

2015

Self Cite

h i g h l i g h t s• We present a formal model to represent the life cycle of data distributed and replicated on many systems.• We leverage this model to propose a programming model that allows users to react to life cycle progression.• We illustrate the approach with examples of applications that we programmed with this model. a b s t r a c tThe Big Data challenge consists in managing, storing, analyzing and visualizing these huge and ever growing data sets to extract sense and knowledge. As the volume of data grows exponentially, the management of these data becomes more complex in proportion. A key point is to handle the complexity of the data life cycle, i.e. the various operations performed on data: transfer, archiving, replication, deletion, etc. Indeed, data-intensive applications span over a large variety of devices and e-infrastructures which implies that many systems are involved in data management and processing. We propose Active Data, a programming model to automate and improve the expressiveness of data management applications. We first define the concept of data life cycle and introduce a formal model that allows to expose data life cycle across heterogeneous systems and infrastructures. The Active Data programming model allows code execution at each stage of the data life cycle: routines provided by programmers are executed when a set of events (creation, replication, transfer, deletion) happen to any data. We implement and evaluate the model with four use cases: a storage cache to Amazon-S3, a cooperative sensor network, an incremental implementation of the MapReduce programming model and automated data provenance tracking across heterogeneous systems. Altogether, these scenarios illustrate the adequateness of the model to program applications that manage distributed and dynamic data sets. We also show that applications that do not leverage on data life cycle can still benefit from Active Data to improve their performances.

“…We analyze the characteristics of hybrid MR runtime environment and design the simulator accordingly. The simulator is based on SimGrid [5] and leverages solutions proposed in the scope of the MapReduce ARN project [6].…”

Section: Introductionmentioning

confidence: 99%

BIGhybrid -- A Toolkit for Simulating MapReduce in Hybrid Infrastructures

Anjos

2014 International Symposium on Computer Architecture and High Performance Computing Workshop

Geyer

2014

Self Cite

Abstract-Cloud computing has increasingly been used as a platform for running large business and data processing applications. Although clouds have become highly popular, when it comes to data processing, the cost of usage is not negligible. Conversely, Desktop Grids, have been used by a plethora of projects, taking advantage of the high number of resources provided for free by volunteers. Merging cloud computing and desktop grids into hybrid infrastructure can provide a feasible low-cost solution for big data analysis. Although frameworks like MapReduce have been conceived to exploit commodity hardware, their use on hybrid infrastructure poses some challenges due to large resource heterogeneity and high churn rate. This study introduces BIGhybrid a toolkit to simulate MapReduce on hybrid environments. The main goal is to provide a framework for developers and system designers to address the issues of hybrid MapReduce. In this paper, we describe the framework which simulates the assembly of two existing middleware: BitDewMapReduce for Desktop Grids and Hadoop-BlobSeer for Cloud Computing. Experimental results included in this work demonstrate the feasibility of our approach.