Enabling High Data Throughput in Desktop Grids through Decentralized Data and Metadata Management: The BlobSeer Approach

Nicolae, Bogdan; Antoniu, Gabriel; Bougé, Luc

doi:10.1007/978-3-642-03869-3_40

Cited by 15 publications

(15 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is crucial in achieving a high aggregated throughput for dataintensive applications, as demonstrated by our previous work [13][14][15].…”

Section: Key Featuresmentioning

confidence: 99%

See 1 more Smart Citation

High Throughput Data-Compression for Cloud Storage

Nicolae

2010

Data Management in Grid and Peer-to-Peer Systems

Self Cite

View full text Add to dashboard Cite

Abstract. As data volumes processed by large-scale distributed dataintensive applications grow at high-speed, an increasing I/O pressure is put on the underlying storage service, which is responsible for data management. One particularly difficult challenge, that the storage service has to deal with, is to sustain a high I/O throughput in spite of heavy access concurrency to massive data. In order to do so, massively parallel data transfers need to be performed, which invariably lead to a high bandwidth utilization. With the emergence of cloud computing, data intensive applications become attractive for a wide public that does not have the resources to maintain expensive large scale distributed infrastructures to run such applications. In this context, minimizing the storage space and bandwidth utilization is highly relevant, as these resources are paid for according to the consumption. This paper evaluates the trade-off resulting from transparently applying data compression to conserve storage space and bandwidth at the cost of slight computational overhead. We aim at reducing the storage space and bandwidth needs with minimal impact on I/O throughput when under heavy access concurrency. Our solution builds on BlobSeer, a highly parallel distributed data management service specifically designed to enable reading, writing and appending huge data sequences that are fragmented and distributed at a large scale. We demonstrate the benefits of our approach by performing extensive experimentations on the Grid'5000 testbed.

show abstract

“…This is crucial in achieving a high aggregated throughput for dataintensive applications, as demonstrated by our previous work [13][14][15].…”

Section: Key Featuresmentioning

confidence: 99%

“…-We apply our proposal to improve BlobSeer [13][14][15], a data management service specifically designed to address the needs of data-intensive applications. -We perform extensive experimentations on the Grid5000 testbed [11] in order to demonstrate the benefits of our approach.…”

Section: Introductionmentioning

confidence: 99%

High Throughput Data-Compression for Cloud Storage

Nicolae

2010

Data Management in Grid and Peer-to-Peer Systems

Self Cite

View full text Add to dashboard Cite

show abstract

“…Second, we propose a generalization for a set of versioning algorithms for data management we initially introduced in [21,22]. We have introduced new data structures and redesigned several aspects to account for better decentralized management, asynchrony, fault tolerance and last but not least allow the user to explicitly control written data layout such that it is optimally distributed for reading.…”

Section: Contributionmentioning

confidence: 99%

BlobSeer: Next-generation data management for large scale infrastructures

Nicolae

Antoniu

Bougé

et al. 2011

Journal of Parallel and Distributed Computing

Self Cite

106

View full text Add to dashboard Cite

As data volumes increase at a high speed in more and more application fields of science, engineering, information services, etc., the challenges posed by data-intensive computing gain an increasing importance. The emergence of highly scalable infrastructures, e.g. for cloud computing and for petascale computing and beyond introduces additional issues for which scalable data management becomes an immediate need. This paper brings several contributions. First, it proposes a set of principles for designing highly scalable distributed storage systems that are optimized for heavy data access concurrency. In particular, we highlight the potentially large benefits of using versioning in this context. Second, based on these principles, we propose a set of versioning algorithms, both for data and metadata, that enable a high throughput under concurrency. Finally, we implement and evaluate these algorithms in the BlobSeer prototype, that we integrate as a storage backend in the Hadoop MapReduce framework. We perform extensive microbenchmarks as well as experiments with real MapReduce applications: they demonstrate that applying the principles defended in our approach brings substantial benefits to data intensive applications.

show abstract

“…The typical size for a page within a blob can be smaller that 1 MB, whence the challenge of dealing with hundreds of thousands of pages belonging to just one BLOB. BlobSeer provides efficient support for heavily-concurrent accesses to the stored data, reaching a throughput of 6.7 GB/s aggregated bandwidth for a configuration with 60 metadata providers, 90 data providers and 360 concurrent writers, as explained in [11].…”

Section: Blobseermentioning

confidence: 99%

“…Dynamic dimensioning: Extensive performance evaluations [11] carried out for BlobSeer reveal that the aggregate bandwidth of concurrent WRITE or READ operations grows as the number of data providers and metadata providers increases. However, deploying BlobSeer's providers on a large number of physical nodes can be an expensive approach, and their optimum number is often unpredictable, as it depends on the load of the providers and on the number of clients concurrently accessing them.…”

Section: A Self-adaptation: What To Adapt?mentioning

confidence: 99%

Bringing Introspection Into the BlobSeer Data-Management System Using the MonALISA Distributed Monitoring Framework

Carpen-Amarie

Cai

Costan

et al. 2010

2010 International Conference on Complex, Intelligent and Software Intensive Systems

Self Cite

View full text Add to dashboard Cite

Abstract-Introspection is the prerequisite of an autonomic behavior, the first step towards a performance improvement and a resource-usage optimization for largescale distributed systems. In grid environments, the task of observing the application behavior is assigned to monitoring systems. However, most of them are designed to provide general resource information and do not consider specific information for higher-level services. More specifically, in the context of data-intensive applications, a specific introspection layer is required in order to collect data about the usage of storage resources, about data access patterns, etc. This paper discusses the requirements for an introspection layer in a data-management system for large-scale distributed infrastructures. We focus on the case of BlobSeer, a large-scale distributed system for storing massive data. The paper explains why and how to enhance BlobSeer with introspective capabilities and proposes a three-layered architecture relying on the MonALISA monitoring framework. This approach has been evaluated on the Grid'5000 testbed, with experiments that prove the feasibility of generating relevant information related to the state and the behavior of the system.

show abstract

Enabling High Data Throughput in Desktop Grids through Decentralized Data and Metadata Management: The BlobSeer Approach

Cited by 15 publications

References 12 publications

High Throughput Data-Compression for Cloud Storage

High Throughput Data-Compression for Cloud Storage

BlobSeer: Next-generation data management for large scale infrastructures

Bringing Introspection Into the BlobSeer Data-Management System Using the MonALISA Distributed Monitoring Framework

Contact Info

Product

Resources

About