BlobSeer: Bringing high throughput under heavy concurrency to Hadoop Map-Reduce applications

Nicolae, Bogdan; Moise, Diana; Antoniu, Gabriel; Bougé, Luc; Dorier, Matthieu

doi:10.1109/ipdps.2010.5470433

Cited by 46 publications

(27 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is crucial in achieving a high aggregated throughput for dataintensive applications, as demonstrated by our previous work [13][14][15].…”

Section: Key Featuresmentioning

confidence: 99%

See 1 more Smart Citation

High Throughput Data-Compression for Cloud Storage

Nicolae

2010

Data Management in Grid and Peer-to-Peer Systems

Self Cite

View full text Add to dashboard Cite

Abstract. As data volumes processed by large-scale distributed dataintensive applications grow at high-speed, an increasing I/O pressure is put on the underlying storage service, which is responsible for data management. One particularly difficult challenge, that the storage service has to deal with, is to sustain a high I/O throughput in spite of heavy access concurrency to massive data. In order to do so, massively parallel data transfers need to be performed, which invariably lead to a high bandwidth utilization. With the emergence of cloud computing, data intensive applications become attractive for a wide public that does not have the resources to maintain expensive large scale distributed infrastructures to run such applications. In this context, minimizing the storage space and bandwidth utilization is highly relevant, as these resources are paid for according to the consumption. This paper evaluates the trade-off resulting from transparently applying data compression to conserve storage space and bandwidth at the cost of slight computational overhead. We aim at reducing the storage space and bandwidth needs with minimal impact on I/O throughput when under heavy access concurrency. Our solution builds on BlobSeer, a highly parallel distributed data management service specifically designed to enable reading, writing and appending huge data sequences that are fragmented and distributed at a large scale. We demonstrate the benefits of our approach by performing extensive experimentations on the Grid'5000 testbed.

show abstract

“…This is crucial in achieving a high aggregated throughput for dataintensive applications, as demonstrated by our previous work [13][14][15].…”

Section: Key Featuresmentioning

confidence: 99%

“…-We apply our proposal to improve BlobSeer [13][14][15], a data management service specifically designed to address the needs of data-intensive applications. -We perform extensive experimentations on the Grid5000 testbed [11] in order to demonstrate the benefits of our approach.…”

Section: Introductionmentioning

confidence: 99%

High Throughput Data-Compression for Cloud Storage

Nicolae

2010

Data Management in Grid and Peer-to-Peer Systems

Self Cite

View full text Add to dashboard Cite

show abstract

“…Moreover, we evaluate BlobSeer as a BLOB-based underlying storage layer for the Hadoop MapReduce framework. The work on integrating BlobSeer with Hadoop, presented in [23] has been used to evaluate the improvements of BlobSeer in the context of MapReduce data-intensive applications. We performed extensive experimentations that demonstrate substantial benefit of applying our principles in this context.…”

Section: Contributionmentioning

confidence: 99%

“…Then we substituted the original data storage layer of Hadoop, Hadoop Distributed File System -HDFS with BSFS. Using this approach (preliminarily described in our previous work [23]), we have now extensively evaluated the impact of our BSFS by performing new experiments both with synthetic microbenchmarks and with real MapReduce applications. The microbenchmarks consist of processes that access the storage layer directly using the file system …”

Section: Blobseer As a File System For Hadoopmentioning

confidence: 99%

BlobSeer: Next-generation data management for large scale infrastructures

Nicolae

Antoniu

Bougé

et al. 2011

Journal of Parallel and Distributed Computing

Self Cite

106

View full text Add to dashboard Cite

As data volumes increase at a high speed in more and more application fields of science, engineering, information services, etc., the challenges posed by data-intensive computing gain an increasing importance. The emergence of highly scalable infrastructures, e.g. for cloud computing and for petascale computing and beyond introduces additional issues for which scalable data management becomes an immediate need. This paper brings several contributions. First, it proposes a set of principles for designing highly scalable distributed storage systems that are optimized for heavy data access concurrency. In particular, we highlight the potentially large benefits of using versioning in this context. Second, based on these principles, we propose a set of versioning algorithms, both for data and metadata, that enable a high throughput under concurrency. Finally, we implement and evaluate these algorithms in the BlobSeer prototype, that we integrate as a storage backend in the Hadoop MapReduce framework. We perform extensive microbenchmarks as well as experiments with real MapReduce applications: they demonstrate that applying the principles defended in our approach brings substantial benefits to data intensive applications.

show abstract

“…It is able to simulate two middleware for two distinct infrastructures: BitDew-MR [7], [8] for Desktop Grid Computing and Hadoop-Blobseer [9] for Cloud computing. BIGhybrid has several desirable features: it is built atop of MRSG, a validated Hadoop simulator [10], and MRA++, a simulator for heterogeneous environments [11]; it has a trace toolkit that enable analyze, monitor and graphically plot the task executions; it is a trace-base simulator able to process real infrastructure availability traces [12]; and its modular design allows for further extension.…”

Section: Introductionmentioning

confidence: 99%

BIGhybrid -- A Toolkit for Simulating MapReduce in Hybrid Infrastructures

Anjos

Fedak

Geyer

2014

2014 International Symposium on Computer Architecture and High Performance Computing Workshop

View full text Add to dashboard Cite

Abstract-Cloud computing has increasingly been used as a platform for running large business and data processing applications. Although clouds have become highly popular, when it comes to data processing, the cost of usage is not negligible. Conversely, Desktop Grids, have been used by a plethora of projects, taking advantage of the high number of resources provided for free by volunteers. Merging cloud computing and desktop grids into hybrid infrastructure can provide a feasible low-cost solution for big data analysis. Although frameworks like MapReduce have been conceived to exploit commodity hardware, their use on hybrid infrastructure poses some challenges due to large resource heterogeneity and high churn rate. This study introduces BIGhybrid a toolkit to simulate MapReduce on hybrid environments. The main goal is to provide a framework for developers and system designers to address the issues of hybrid MapReduce. In this paper, we describe the framework which simulates the assembly of two existing middleware: BitDewMapReduce for Desktop Grids and Hadoop-BlobSeer for Cloud Computing. Experimental results included in this work demonstrate the feasibility of our approach.

show abstract

BlobSeer: Bringing high throughput under heavy concurrency to Hadoop Map-Reduce applications

Cited by 46 publications

References 6 publications

High Throughput Data-Compression for Cloud Storage

High Throughput Data-Compression for Cloud Storage

BlobSeer: Next-generation data management for large scale infrastructures

BIGhybrid -- A Toolkit for Simulating MapReduce in Hybrid Infrastructures

Contact Info

Product

Resources

About