Distributed data storage systems with opportunistic repair

Aggarwal, Vaneet; Tian, Chao; Vaishampayan, Vinay A.; Chen, Yih-Farn Robin

doi:10.1109/infocom.2014.6848122

Cited by 15 publications

(11 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We mostly focus on one dimensional MDS protection schemes in this study, however it can be shown that the model can be used to derive closed form expressions for multi dimensional MDS disk arrays [21] and code structures [22]. Furthermore, although it is out of scope of this paper, we can show that the general model proposed in this study can be used to predict reliability for non-MDS codes [23], [24], array BP-XOR codes [25], [26] as well as repair-efficient codes [27], [28] under novel opportunistic repair mechanisms [29].…”

Section: A Related Workmentioning

confidence: 89%

A Reliability Model for Dependent and Distributed MDS Disk Array Units

Arslan

2019

IEEE Trans. Rel.

View full text Add to dashboard Cite

Archiving and systematic backup of large digital data generates a quick demand for multi-peta byte scale storage systems. As drive capacities continue to grow beyond the few terabytes range to address the demands of today's cloud, the likelihood of having multiple/simultaneous disk failures become a reality. Among the main factors causing catastrophic system failures, correlated disk failures and the network bandwidth are reported to be the two common source of performance degradation. The emerging trend is to use efficient/sophisticated erasure codes (EC) equipped with multiple parities and efficient repairs in order to meet the reliability/bandwidth requirements. It is known that mean time to failure and repair rates reported by the disk manufacturers cannot capture life cycle patterns of distributed storage systems. In this study, we develop failure models based on generalized Markov chains that can accurately capture correlated performance degradations with multi-parity protection schemes based on modern Maximum Distance Separable (MDS) EC. Furthermore, we use the proposed model in a distributed storage scenario to quantify two example use cases: Primarily, the common sense that adding more parity disks are only meaningful if we have a decent decorrelation between the failure domains of storage systems and the reliability of generic multiple singledimensional EC protected storage systems.

show abstract

Section: A Related Workmentioning

confidence: 89%

A Reliability Model for Dependent and Distributed MDS Disk Array Units

Arslan

2019

IEEE Trans. Rel.

View full text Add to dashboard Cite

show abstract

“…al. [18] analysed the mean-time-to-failure (MTTF) in the regenerating codes with and without bandwidth adaptivity. This analysis is based on a birth-death process model in which the population of available storage node randomly changes with appropriately chosen rates.…”

Section: B Background On Bandwidth Adaptive Repairmentioning

confidence: 99%

Bandwidth Adaptive & Error Resilient MBR Exact Repair Regenerating Codes

Mahdaviani

Khisti

Mohajer

2019

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

Regenerating codes are efficient methods for distributed storage in storage networks, where node failures are common. They guarantee low cost data reconstruction and repair through accessing only a predefined number of arbitrarily chosen storage nodes in the network. In this work we consider two simultaneous extensions to the original regenerating codes framework introduced in [1]; i) both data reconstruction and repair are resilient to the presence of a certain number of erroneous nodes in the network and ii) the number of helper nodes in every repair is not fixed, but is a flexible parameter that can be selected during the runtime. We study the fundamental limits of required total repair bandwidth and provide an upper bound for the storage capacity of these codes under these assumptions. We then focus on the minimum repair bandwidth (MBR) case and derive the exact storage capacity by presenting explicit coding schemes with exact repair, which achieve the upper bound of the storage capacity in the considered setup. To this end, we first provide a more natural extension of the well-known Product Matrix (PM) MBR codes [2], modified to provide flexibility in the choice of number of helpers in each repair, and simultaneously be robust to erroneous nodes in the network. This is achieved by proving the non-singularity of family of matrices in large enough finite fields. We next provide another extension of the PM codes, based on novel repair schemes which enable flexibility in the number of helpers and robustness against erroneous nodes without any extra cost in field size compared to the original PM codes. 1

show abstract

“…al. [29] also considered a functional repair setup to achieve bandwidth adaptivity in regenerating codes. They analysed the mean-time-to-Data-Loss (MTTDL) in the regenerating codes with and without bandwidth adaptivity.…”

Section: Related Workmentioning

confidence: 99%

“…Remark 3. Note that calculating the inverse matrices (Ω H (i)) −1 , for i ∈ {1, · · · , β}, can be carried out recursively since from (29) we have,…”

mentioning

confidence: 99%

Product Matrix MSR Codes With Bandwidth Adaptive Exact Repair

Mahdaviani

Mohajer

Khisti

2018

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

In a distributed storage systems (DSS) with k systematic nodes, robustness against node failure is commonly provided by storing redundancy in a number of other nodes and performing repair mechanism to reproduce the content of the failed nodes. Efficiency is then achieved by minimizing the storage overhead and the amount of data transmission required for data reconstruction and repair, provided by coding solutions such as regenerating codes [1]. Common explicit regenerating code constructions enable efficient repair through accessing a predefined number, d, of arbitrary chosen available nodes, namely helpers. In practice, however, the state of the system dynamically changes based on the request load, the link traffic, etc., and the parameters which optimize system's performance vary accordingly. It is then desirable to have coding schemes which are able to operate optimally under a range of different parameters simultaneously. Specifically, adaptivity in the number of helper nodes for repair is of interest. While robustness requires capability of performing repair with small number of helpers, it is desirable to use as many helpers as available to reduce the transmission delay and total repair traffic.In this work we focus on the minimum storage regenerating (MSR) codes, where each of the n nodes in the network is supposed to store α information units, and the source data of size kα could be recovered from any arbitrary set of k nodes. We introduce a class of MSR codes that realize optimal repair bandwidth simultaneously with a set of different choices for the number of helpers, namely D = {d1, · · · , d δ }. Our coding scheme follows the Product Matrix (PM) framework introduced in [2], and could be considered as a generalization of the PM MSR code presented in [2], such that any di = (i + 1)(k − 1) helpers can perform an optimal repair. As a result, the coding rate in our construction is limited by k n ≤ 1 2 . However, similar to the original design of PM MSR codes, our solution can realize practical values of the parameter α. Recently [3] has presented another explicit MSR coding scheme which is capable of performing optimal repair for various number of helpers. The solution presented in [3] works for any arbitrary set of parameters k, D and can achieve high coding rates, but the required α for this code is exponentially large. We show that the required value for α in the coding scheme presented in this work is exponentially smaller when compared to [3] for the same set of other parameters. Particularly, for a DSS with n nodes, and k systematic nodes, the required value for α is reduced from s n to sk, where s = lcm (d1 − k + 1, · · · , d δ − k + 1). We also show the required field size in the presented coding scheme is equal to n 1 .

show abstract

Distributed data storage systems with opportunistic repair

Cited by 15 publications

References 21 publications

A Reliability Model for Dependent and Distributed MDS Disk Array Units

A Reliability Model for Dependent and Distributed MDS Disk Array Units

Bandwidth Adaptive & Error Resilient MBR Exact Repair Regenerating Codes

Product Matrix MSR Codes With Bandwidth Adaptive Exact Repair

Contact Info

Product

Resources

About