Differential Erasure Codes for Efficient Archival of Versioned Data in Cloud Storage Systems

Abstract

In this paper, we study the problem of storing an archive of versioned data in a reliable and efficient manner in distributed storage systems. We propose a new storage technique called differential erasure coding (DEC) where the differences (deltas) between subsequent versions are stored rather than the whole objects, akin to a typical delta encoding technique. However, unlike delta encoding techniques, DEC opportunistically exploits the sparsity (i.e., when the differences between two successive versions have few non-zero entries) in the updates to store the deltas using compressed sensing techniques applied with erasure coding. We first show that DEC provides significant savings in the storage size for versioned data whenever the update patterns are characterized by in-place alterations. Subsequently, we propose a practical DEC framework so as to reap storage size benefits against not just in-place alterations but also real-world update patterns such as insertions and deletions that alter the overall data sizes. We conduct experiments with several synthetic workloads to demonstrate that the practical variant of DEC provides significant reductions in storage overhead (up to 60% depending on the workload) compared to baseline storage system which incorporates concepts from Rsync, a delta encoding technique to store and synchronize data across a network.

Multi-Version Coding—An Information-Theoretic Perspective of Consistent Distributed Storage

“…We note that the idea of storing versioned data has acquired some recent interest in information theory literature. In particular, some of the challenges of updating data in distributed storage systems have been studied in [14], [15], [16], [17]. These works complement our paper, and their ideas can perhaps be adapted to our framework to build efficient consistent data storage implementations.…”

“…Refinements of multi-version coding, and our toy model for channels to incorporate these requirements is an important direction of future work. In particular, the tools used in references [14], [15], [16], [17], when appropriately adopted to multi-version coding, may help reduce latency by reducing the amount of information transmitted to disperse and update information related to a new version.…”

“…Developing code constructions that exploit dependency in the versions is an interesting area of future work. The ideas of [14], [16] can be useful in this endeavor. • The framework of our toy model can be developed to study more realistic scenarios.…”

“…, S(A t ). That is Property (7) states that the above set can be computed using Y [At−1] , Z t , W T −{Π(t)} via equation (16). Therefore, we can compute W Π(t) as…”

On the Adoption of Erasure Code for Cloud Storage by Major Distributed Storage Systems

“…Update or mutation can happen beyond a single block, and this further entails additional processing. Authors of [62], [63], [64] and [65] have had significant enhancements to implement an updatable erasure coded DSS.…”