AZ-Code: An Efficient Availability Zone Level Erasure Code to Provide High Fault Tolerance in Cloud Storage Systems

Xie, Xin; Wu, Chongqing; Gu, Jianlei; Qiu, Han; Li, Jie; Guo, Minyi; He, Xubin; Dong, Yuanyuan; Zhao, Yafei

doi:10.1109/msst.2019.00004

Cited by 29 publications

(15 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several papers have studied the design of in-memory key-value stores [56,58,46,82,76,22,73,5,26]. A significant body of work focuses on minimizing repair costs and encoding/decoding [68,72,18,77,42,50,64,75,79,40,41,26]. The cost savings offered by EC have motivated its use particularly in production archival (i.e., writeonce/rarely) storage systems [37,51].…”

Section: Related Workmentioning

confidence: 99%

LEGOStore: A Linearizable Geo-Distributed Store Combining Replication and Erasure Coding

Zare¹,

Cadambe²,

Urgaonkar³

et al. 2021

Preprint

View full text Add to dashboard Cite

We design and implement LEGOStore, an erasure coding (EC) based linearizable data store over geo-distributed public cloud data centers (DCs). For such a data store, the confluence of the following factors opens up opportunities for EC to be latency-competitive with replication: (a) the necessity of communicating with remote DCs to tolerate entire DC failures and implement linearizability; and (b) the emergence of DCs near most large population centers. LEGOStore employs an optimization framework that, for a given object, carefully chooses among replication and EC, as well as among various DC placements to minimize overall costs. To handle workload dynamism, LEGOStore employs a novel agile reconfiguration protocol. Our evaluation using a LEGOStore prototype spanning 9 Google Cloud Platform DCs demonstrates the efficacy of our ideas. We observe cost savings ranging from moderate (5-20%) to significant (60%) over baselines representing the state of the art while meeting tail latency SLOs. Our reconfiguration protocol is able to transition key placements in 3 to 4 inter-DC RTTs (< 1s in our experiments), allowing for agile adaptation to dynamic conditions .

show abstract

Section: Related Workmentioning

confidence: 99%

LEGOStore: A Linearizable Geo-Distributed Store Combining Replication and Erasure Coding

Zare¹,

Cadambe²,

Urgaonkar³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…It is known that popular erasure codes can be divided into 2 classes: RS-based codes and XOR-based codes [62]. Accordingly, we will discuss RS-based DU and XOR-based DU.…”

Section: Existing Classification a An Example Of Dumentioning

confidence: 99%

“…Taking QFS (Quantcast File System) [38] as an example, the implementation of QFS with erasure coding can save 50% of storage overhead over the original HDFS which uses 3-replication, while tolerating the same number of failures. The property of low storage overhead on erasure coding is so attractive for many cloud computing providers that erasure coding is gradually employed by many leading players in cloud storage, such as Amazon S3 [5], Google cloud [23], Microsoft Azure [28], Facebook cluster [43] and Alibaba Cloud [62].…”

Section: Introductionmentioning

confidence: 99%

Erasure Coding-Oriented Data Update for Cloud Storage: A Survey

2020

View full text Add to dashboard Cite

Erasure coding is the leading technique to achieve resilient redundancy in cloud storage systems. However, it introduces two prominent issues: data repair and data update. Compare to data repair, data update is much more common. A variety of update schemes based on erasure coding have been proposed in the literature to optimize data update, such as computation optimization, network traffic overhead reduction, IO overhead reduction, and modern hardware acceleration. However, all of these techniques were proposed individually previously. In this work, we seek to summarize them systematically and group them in a new form. First, we generalize the state-of-the-art researches and introduce existing classifications. Moreover, based on our observation, we propose two classifications: resource-based classification and tierbased classification. In resource-based classification, we group these techniques according to the resource they optimize and introduce them in detail. In tier-based classification, we propose a novel hybrid technique framework with five tiers and conduct a comprehensive comparison between these techniques. We make a conjecture that most techniques in different tiers can be used jointly. Finally, we conclude the research challenges and potential future works. INDEX TERMS data update, cloud storage, erasure coding, survey

show abstract

“…These approaches are combined with several classic erasure codes such as STAR Huang and Xu (2008), RS6 Zhang et al (2015), Gu et al (2019), TIP Zhang et al (2015), etc. and other varies MDS codes Reed and Solomon (1960), Corbett et al (2004), Plank (2008) in RAID-6, or codes like AZ-Code Xie et al (2019) and Approximate Code Jin et al (2019), etc.. To adapt the dynamic change on application workloads, CRAID Miranda and Cortes (2014) is a general approach for disk arrays. Some other adaptive industrial products integrated with scaling approaches are RAID-x Hwang et al (2000) and AutoRAID Wilkes (1996).…”

Section: Existing Scaling Approaches In Raidmentioning

confidence: 99%

AIR: an approximate intelligent redistribution approach to accelerate RAID scaling

Lin

Guo

2020

CCF Trans. HPC

Self Cite

View full text Add to dashboard Cite

Nowadays, videos and images are becoming the predominant format of data storage, which take up more space than conventional plain texts. This rapid increment brings a high requirement on scalability in large data centers, where disk arrays (also referred to as "RAID") are the main devices to store the numerous data. To improve the scalability of RAID systems, several scaling approaches are proposed to guarantee a uniform data distribution, such as decreasing migration I/Os and speeding up the scaling process. However, typical approaches are offline and ignore the impacts of concurrent application I/Os, which plays an important role on RAID scaling (i.e., data migration, data distribution, etc.). For example, an exact uniform data distribution among disks doesn't mean an even I/O accesses to these disks. To address this problem, in this paper, we propose an approximate intelligent redistribution (AIR) approach to accelerate RAID scaling. The main idea of AIR is utilizing the dynamic data access patterns from concurrent application workloads, and providing an approximate data distribution to guarantee a uniform I/O accesses to various data disks. To achieve this goal, AIR utilizes the prevailing machine learning algorithms to identify hot data from application workloads, and gives an intelligent migration approach to minimize the data movements. By this way, AIR can sharply cut down the migration I/Os. To demonstrate the effectiveness of AIR approach, we conduct several simulations via Disksim. The results show that, compared to traditional RAID scaling approaches such as FastScale and GSR, AIR saves up to 99.3% I/O cost and reduces the data migration ratio by up to 95.8%, which speeds up the scaling process by a factor of up to 30.3X.

show abstract

AZ-Code: An Efficient Availability Zone Level Erasure Code to Provide High Fault Tolerance in Cloud Storage Systems

Cited by 29 publications

References 29 publications

LEGOStore: A Linearizable Geo-Distributed Store Combining Replication and Erasure Coding

LEGOStore: A Linearizable Geo-Distributed Store Combining Replication and Erasure Coding

Erasure Coding-Oriented Data Update for Cloud Storage: A Survey

AIR: an approximate intelligent redistribution approach to accelerate RAID scaling

Contact Info

Product

Resources

About