Access-optimal Linear MDS Convertible Codes for All Parameters

Maturana, Francisco; Mukka, V. S. Chaitanya; Rashmi, K. V.

doi:10.1109/isit44484.2020.9173947

Cited by 17 publications

(18 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are explicit constructions [6], [7] of access-optimal convertible codes for all valid parameters (n I , k I ; n F , k F ). Notice that for Regime 1 (r I < r F ), read access cost is always M , which is the same as the default approach.…”

Section: Theorem 1 ([7]mentioning

confidence: 99%

“…In Section II-B, we reviewed the definition of convertible codes from literature [6], [7]. Existing works on convertible codes [6], [7] have considered only scalar codes, where each code symbol corresponds to a scalar from a finite field F q . Considering scalar codes is sufficient when optimizing for access cost, which was the focus in these prior works, since the access cost is measured at the granularity of code symbols.…”

Section: Modeling Conversion For Optimizing Network Bandwidthmentioning

confidence: 99%

“…Since the maximum supported r F is 3, we start with an access-optimal (7, 4; 11, 8) convertible code. Thus, C I is a [7,4] code, C F is a [11,8] code, and C I is a [5,4] code. In the first round of piggybacking we consider r F = 2, which yields the code shown in Example 1.…”

Section: B Convertible Codes With Bandwidth-optimal Conversion For Mu...mentioning

confidence: 99%

“…For the access cost of conversion in the merge regime, it is known [7] that one cannot do better than the default approach for a wide range of parameters (specifically, when (n I − k I ) < (n F − k F ), which we term Regime 1). For the remaining set of parameters (which we term Regime 2), access-optimal convertible codes lead to considerable reduction in access cost compared to the default approach.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Bandwidth Cost of Code Conversions in Distributed Storage: Fundamental Limits and Optimal Constructions

Maturana¹,

Rashmi²

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Erasure codes have become an integral part of distributed storage systems as a tool for providing data reliability and durability under the constant threat of device failures. In such systems, an [n, k] code over a finite field F q encodes k message symbols from F q into n codeword symbols from F q which are then stored on n different nodes in the system. Recent work has shown that significant savings in storage space can be obtained by tuning n and k to variations in device failure rates. Such a tuning necessitates code conversion: the process of converting already encoded data under an initial [n I , k I ] code to its equivalent under a final [n F , k F ] code. The default approach to conversion is to re-encode the data under the new code, which places significant burden on system resources. Convertible codes are a recently proposed class of codes for enabling resource-efficient conversions. Existing work on convertible codes has focused on minimizing the access cost, i.e., the number of code symbols accessed during conversion. Bandwidth, which corresponds to the amount of data read and transferred, is another important resource to optimize during conversions.In this paper, we initiate the study on the fundamental limits on bandwidth used during code conversion and present constructions for bandwidth-optimal convertible codes. First, we model the code conversion problem using network information flow graphs with variable capacity edges. Second, focusing on MDS codes and an important parameter regime called the merge regime, we derive tight lower bounds on the bandwidth cost of conversion. The derived bounds show that the bandwidth cost of conversion can be significantly reduced even in regimes where it has been shown that access cost cannot be reduced as compared to the default approach. Third, we present a new construction for MDS convertible codes which matches the proposed lower bound and is thus bandwidth-optimal during conversion. 1 In the literature, this set of n symbols is sometimes called a stripe instead of a codeword. In this work, we make no distinctions between these two terms.

show abstract

Section: Theorem 1 ([7]mentioning

confidence: 99%

Section: Modeling Conversion For Optimizing Network Bandwidthmentioning

confidence: 99%

Section: B Convertible Codes With Bandwidth-optimal Conversion For Mu...mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Bandwidth Cost of Code Conversions in Distributed Storage: Fundamental Limits and Optimal Constructions

Maturana¹,

Rashmi²

2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Second, file sizes often change in storage systems with frequent writes [14]. In this case, the entire file has to be encoded again [15]. Third, to be able to code V fragments of a file into V R MDS coded symbols, the symbols must belong to a sufficiently large alphabet [16,Theorem 4.1].…”

Section: Introductionmentioning

confidence: 99%

Latency optimal storage and scheduling of replicated fragments for memory-constrained servers

Jinan,

Badita,

Sarvepalli

et al. 2020

Preprint

View full text Add to dashboard Cite

We consider the setting of distributed storage system where a single file is subdivided into smaller fragments of same size which are then replicated with a common replication factor across servers of identical cache size. An incoming file download request is sent to all the servers, and the download is completed whenever request gathers all the fragments. At each server, we are interested in determining the set of fragments to be stored, and the sequence in which fragments should be accessed, such that the mean file download time for a request is minimized. We model the fragment download time as an exponential random variable independent and identically distributed for all fragments across all servers, and show that the mean file download time can be lower bounded in terms of the expected number of useful servers summed over all distinct fragment downloads. We present deterministic storage schemes that attempt to maximize the number of useful servers. We show that finding the optimal sequence of accessing the fragments is a Markov decision problem, whose complexity grows exponentially with the number of fragments. We propose heuristic algorithms that determine the sequence of access to the fragments which are empirically shown to perform well.

show abstract

Storage Reliability

Shu

2024

Data Storage Architectures and Technologies

View full text Add to dashboard Cite

Access-optimal Linear MDS Convertible Codes for All Parameters

Cited by 17 publications

References 24 publications

Bandwidth Cost of Code Conversions in Distributed Storage: Fundamental Limits and Optimal Constructions

Bandwidth Cost of Code Conversions in Distributed Storage: Fundamental Limits and Optimal Constructions

Latency optimal storage and scheduling of replicated fragments for memory-constrained servers

Storage Reliability

Contact Info

Product

Resources

About