High Throughput Data-Compression for Cloud Storage

Abstract

Abstract. As data volumes processed by large-scale distributed dataintensive applications grow at high-speed, an increasing I/O pressure is put on the underlying storage service, which is responsible for data management. One particularly difficult challenge, that the storage service has to deal with, is to sustain a high I/O throughput in spite of heavy access concurrency to massive data. In order to do so, massively parallel data transfers need to be performed, which invariably lead to a high bandwidth utilization. With the emergence of cloud computing, data intensive applications become attractive for a wide public that does not have the resources to maintain expensive large scale distributed infrastructures to run such applications. In this context, minimizing the storage space and bandwidth utilization is highly relevant, as these resources are paid for according to the consumption. This paper evaluates the trade-off resulting from transparently applying data compression to conserve storage space and bandwidth at the cost of slight computational overhead. We aim at reducing the storage space and bandwidth needs with minimal impact on I/O throughput when under heavy access concurrency. Our solution builds on BlobSeer, a highly parallel distributed data management service specifically designed to enable reading, writing and appending huge data sequences that are fragmented and distributed at a large scale. We demonstrate the benefits of our approach by performing extensive experimentations on the Grid'5000 testbed.

On the Benefits of Transparent Compression for Cost-Effective Cloud Data Storage

“…This contribution extends our previous proposal presented in [18]. In particular, we introduce a more generic compression layer that extends the applicability of our proposal to the management of virtual machine images (in addition to the management of application data) and show how to integrate it in the cloud architecture.…”

“…In particular, in addition to the experiments presented in [18], we highlight the benefits of our approach for virtual machine image storage.…”

An approach of fast data manipulation in HDFS with supplementary mechanisms

“…Chen et al [3] developed a decision algorithm that helps Map/Reduce users identify when and where to use compression. Nicolae [13] evaluated the trade-off resulting from transparently applying data compression to conserve storage space and bandwidth at the cost of slight computational overhead. Crume et al [5] presented a slightly modified version of Hadoop designed for processing scientific data using multiple lossless approaches to compress intermediate the output data of Map/Reduce.…”

Bulut Bi̇li̇şi̇m Si̇stemleri̇nde Veri̇ni̇n Farkli Boyutlari Üzeri̇ne Derleme

“…Veri sıkıştırma, verilerin taşınması ve depolanması aşamalarında daha az yer kaplaması için tercih edilen bir veri yapısı yaklaşımıdır [63][64][65]. Bulut depolama alanları, neredeyse sınırsız bir şekilde kaynak tahsis etmesinden dolayı, verinin depolanmasında sıkıştırılmasına ihtiyaç duyulmamaktadır.…”