A Reliability Model for Dependent and Distributed MDS Disk Array Units

Abstract

Archiving and systematic backup of large digital data generates a quick demand for multi-peta byte scale storage systems. As drive capacities continue to grow beyond the few terabytes range to address the demands of today's cloud, the likelihood of having multiple/simultaneous disk failures become a reality. Among the main factors causing catastrophic system failures, correlated disk failures and the network bandwidth are reported to be the two common source of performance degradation. The emerging trend is to use efficient/sophisticated erasure codes (EC) equipped with multiple parities and efficient repairs in order to meet the reliability/bandwidth requirements. It is known that mean time to failure and repair rates reported by the disk manufacturers cannot capture life cycle patterns of distributed storage systems. In this study, we develop failure models based on generalized Markov chains that can accurately capture correlated performance degradations with multi-parity protection schemes based on modern Maximum Distance Separable (MDS) EC. Furthermore, we use the proposed model in a distributed storage scenario to quantify two example use cases: Primarily, the common sense that adding more parity disks are only meaningful if we have a decent decorrelation between the failure domains of storage systems and the reliability of generic multiple singledimensional EC protected storage systems.

On the Distribution Modeling of Heavy-Tailed Disk Failure Lifetime in Big Data Centers

“…However in the literature, there are still inefficiencies in many popular prediction models of real-life. These deficiencies result in increased amount of data storage due to the utilization of replication and/or erasure codes [18] and associated Markov models for reliability prediction [19]. However, data protection boosting methods through excessive redundancy (especially for replication) may lead to inefficient use of storage resources.…”

“…using MTTF/MTTDL and known distributions, we can simply generate an answer to the probability of failure for the first few years pretty accurately. Plus, closed-form expressions for MTTDL are shown to be possible in [19] for more complicated general cases and such analytic expressions usually help our intuition for modeling error-tolerant data storage systems.…”

Towards Building Reliable and Cost-Efficient Distributed Storage Systems

“…They combine Hier and Flat with the same data redundancy scheme (RS or MSR),and then compare their effects. Arslan [49] uses the Markov model to study the reliability of disk arrays under different Maximum Distance Separable (MDS) erasure codes, different data allocations and different repair rates. Node and rack failures are not included because the study is about the reliability of disk arrays but not distributed storage systems, and no discussions about the combination of different schemes.…”

Cloud2HDD: Large-Scale HDD Data Analysis on Cloud for Cloud Datacenters

“…Many techniques have been proposed in lietrature to improve cloud storage systems against such failures such as information fragmentation [2], and reinforcement learning [3]. Recently, data modeling attempts are made in order to incorporate more drive-related parameters and failure data in the durability prediction of arrays of hard drives [4], [5]. On the other hand, the methods and technology used by manufacturers during the building process can create a faulty connection between distinct storage devices.…”