On the Adoption of Erasure Code for Cloud Storage by Major Distributed Storage Systems

Abstract

INTRODUCTION: Traditional Cloud Systems are struggling to cope with the exponential growth of data in todays' distributed application environment. The amount of data online has continuously increased since 2003. From an estimated 5 Exabyte in 2003 to 988 Exabyte in 2010. Presently it is estimated that 5 Exabyte of data are produced daily. To cope with such astronomical load of data, Distributed Storage Systems such as Amazon, Google and Microsoft Azure are becoming the de-facto method for the storage of data. Replication is the method used for providing redundancy. However Erasure Coding is a worthy alternative. OBJECTIVES: The purpose of this paper is to assess the most used distributed storage systems using different evaluation criteria and identifying how erasure code can be integrated into them. METHODS: This paper provides a survey of well-known Distributed Storage Systems by using the CAP (Consistency, Availability and Partition Tolerance) Theorem. We go by presenting the solution according to the objectives set and tradeoff acknowledged by the designers. RESULTS: A comprehensive survey is presented using five evaluation criteria (design principle, data model, failure detection and recovery, consistency and security). Adoption of erasure code in Distributed Storage Systems is discussed and its advantages are deliberated. Several open challenges are also put forward. CONCLUSION: This paper provides researchers in the field with a comprehensive review of Distributed Storage Systems and how the adoption of erasure codes will enhance their capabilities.

Symmetrical Data Recovery: FPGA-Based Multi-Dimensional Elastic Recovery Acceleration for Multiple Block Failures in Ceph Systems

“…Erasure coding (EC), which originated in the 1960s, is an encoding method designed to enhance the fault tolerance of data transmission [23]. Initially developed to address data loss issues during network transmission in the communications field [24], its core idea involves subdividing the transmission signal and incorporating a verification mechanism, thereby creating interconnections among the signals.…”

Some Specific Features in the Construction of P-Ary Reed-Solomon Codes for an Arbitrary Prime P

“…Arithmetic in GF(p n ) involves polynomial arithmetic modulo the irreducible polynomial p(x). The two main algebraic operations are addition (13) and multiplication (14), which are defined for two elements f(α) (11) and g(α) (12) of GF(p n ).…”

“…Erasure-coded storage using Reed-Solomon codes is now widely used in large, distributed storage systems, including Google File System (GFS), Facebook Hadoop Distributed File System (HDFS), Windows Azure storage, and data centers [9], [10], [11].…”

Robust-DSN: A Hybrid Distributed Replication and Encoding Network Grouped with a Distributed Swarm Workflow Scheduler

“…Moreover, the heterogeneous nature of storage nodes in a distributed environment makes the process more complex for reliable data persistence and fault tolerance. There should be an appropriate approach to analyze the inherent trade-off between performance and storage overhead for effective utilization of the system resources [11,12]. Therefore, a comprehensive solution needs to improve both aspects at the same time in distributed storage systems.…”