Universally Decodable Matrices for Distributed Matrix-Vector Multiplication

Ramamoorthy, Aditya; Tang, Li; Vontobel, Pascal O.

doi:10.1109/isit.2019.8849451

Cited by 47 publications

(34 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our result of Theorem 5.1 also indicates that choosing H = G (m,P ) (ρ (P ) ), i.e., to be a Chebyshev Vandermonde matrix, naturally provides a well-conditioned solution to this problem. Another solution for the matrix-vector multiplication problem is provided in [25] via universally decodable matrices [32]; in this work numerical stability is demonstrated empirically.…”

Section: Discussionmentioning

confidence: 99%

“…It is, however, important to note that the problems resolved in our paper here are more restrictive since matrix multiplication codes -where both matrices are to be encoded so that the product can be recovered -require much more structure than matrix-multiplication where only one matrix is to be encoded. For instance, random Gaussian encoding does not naturally work for matrix multiplication to get a recovery threshold of 2m − 1, and it is not clear whether the solution of [25] is applicable either. The utility of Chebyshev-Vandermonde matrices for a variety of coded computing problems including matrix-vector multiplication, matrix multiplication and Lagrange coded computing motivates the study of low-complexity decoding and error correction mechanisms for these systems.…”

Section: Discussionmentioning

confidence: 99%

“…Using the property of the Chebyshev polynomials that for any i, j ∈ N, T i (x)T j (x) = 1/2 (T i+j (x) + T |i−j| (x)), (25) can be rewritten as…”

Section: A System Model and Problem Formulationmentioning

confidence: 99%

See 2 more Smart Citations

Numerically Stable Polynomially Coded Computing

Fahim

Cadambe

2019

2019 IEEE International Symposium on Information Theory (ISIT)

View full text Add to dashboard Cite

We study the numerical stability of polynomial based encoding methods, which has emerged to be a powerful class of techniques for providing straggler and fault tolerance in the area of coded computing. Our contributions are as follows: 1) We construct new codes for matrix multiplication that achieve the same fault/straggler tolerance as the previously constructed MatDot Codes and Polynomial Codes. Unlike previous codes that use polynomials expanded in a monomial basis, our codes use a basis of orthogonal polynomials. 2) We show that the condition number of every m×m sub-matrix of an m×n, n ≥ m Chebyshev-Vandermonde matrix, evaluated on the n-point Chebyshev grid, grows as O(n 2(n−m) ) for n > m. An implication of this result is that, when Chebyshev-Vandermonde matrices are used for coded computing, for a fixed number of redundant nodes s = n − m, the condition number grows at most polynomially in the number of nodes n. 3) By specializing our orthogonal polynomial based constructions to Chebyshev polynomials, and using our condition number bound for Chebyshev-Vandermonde matrices, we construct new numerically stable techniques for coded matrix multiplication. We empirically demonstrate that our constructions have significantly lower numerical errors compared to previous approaches which involve inversion of Vandermonde matrices. We generalize our constructions to explore the trade-off between computation/communication and fault-tolerance. 4) We propose a numerically stable specialization of Lagrange coded computing. Motivated by our condition number bound, our approach involves the choice of evaluation points and a suitable decoding procedure that involves inversion of an appropriate Chebyshev-Vandermonde matrix. Our approach is demonstrated empirically to have lower numerical errors as compared to standard methods.M. Fahim and V. Cadambe are with the 1 For example, [22], reports that "In our experiments we observed large floating point errors when inverting high degree Vandermonde matrices for polynomial interpolation". 2 The master and fusion nodes are logical entities; in practice, they may be the same node, or may be emulated in a decentralized manner by the computation nodes.A simple generalization of the above example, described in Construction 1 in Section IV, leads to a class of codes, we refer to it as OrthoMatDot Codes, with recovery threshold of 2m − 1, the same recovery threshold as MatDot Codes. In general, orthonormal polynomials are defined over arbitrary weight measure 1 −1 · w(x)dx; some well known classes of polynomials corresponding to different weight measures w(x) include Legendre, Chebyshev, Jacobi and Laguerre Polynomials [20], [21] (See Section III for definitions). Our OrthoMatDot Codes in Section IV can use any weight measure, and therefore can be used with different classes of orthonormal polynomials. Of particular interest to our paper are the Chebyshev polynomials ( Fig. 4).With our basic template, the task of developing numerically stable codes boils down to (A) interpolating p A (x)p...

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Numerically Stable Polynomially Coded Computing

Fahim

Cadambe

2019

2019 IEEE International Symposium on Information Theory (ISIT)

View full text Add to dashboard Cite

show abstract

“…In [22] each worker is tasked by a specified fraction of coded and uncoded computations. In [23] multiple coded subtasks assigned to each worker are generated according to the characteristics of universally decodable matrices. In [24] each worker is tasked with completing a fully-uncoded series of subtasks with respect to a predesigned computation order.…”

Section: A Background: Stragglers and Coded Computingmentioning

confidence: 99%

Hierarchical coded matrix multiplication

Kiani

Ferdinand

Draper

2019

2019 16th Canadian Workshop on Information Theory (CWIT)

View full text Add to dashboard Cite

In distributed computing systems slow working nodes, known as stragglers, can greatly extend finishing times. Coded computing is a technique that enables straggler-resistant computation. Most coded computing techniques presented to date provide robustness by ensuring that the time to finish depends only on a set of the fastest nodes. However, while stragglers do compute less work than non-stragglers, in real-world commercial cloud computing systems (e.g., Amazon's Elastic Compute Cloud (EC2)) the distinction is often a soft one. In this paper, we develop hierarchical coded computing that exploits the work completed by all nodes, both fast and slow, automatically integrating the potential contribution of each. We first present a conceptual framework to represent the division of work amongst nodes in coded matrix multiplication as a cuboid partitioning problem. This framework allows us to unify existing methods and motivates new techniques. We then develop three methods of hierarchical coded computing that we term bit-interleaved coded computation (BICC), multilevel coded computation (MLCC), and hybrid hierarchical coded computation (HHCC). In this paradigm, each worker is tasked with completing a sequence (a hierarchy) of ordered subtasks. The sequence of subtasks, and the complexity of each, is designed so that partial work completed by stragglers can be used in, rather than ignored. We note that our methods can be used in conjunction with any coded computing method. We illustrate this showing how we can use our methods to accelerate all previously developed coded computing technique by enabling them to exploit stragglers. Under a widely studied statistical model of completion time, our approach realizes a 66% improvement in expected finishing time. On Amazon EC2, the gain was 28% when stragglers are simulated.

show abstract

“…For simplifying the notation, hereafter, we use ρ N − K − t. Suppose that t ≤ t max = L L+1 (N − K) errors occur at positions j 1 , j 2 , · · · , j t with values e (l) j1 , e (l) j2 , · · · , e (l) jt for the lth RS code. Recall the syndrome matrix S (l) (t) for the lth RS code (see (4)). As shown in [12], S (l) (t) can be decomposed as…”

Section: A Probability Of Failurementioning

confidence: 99%

Collaborative Decoding of Polynomial Codes for Distributed Computation

Subramaniam

Heidarzadeh

Narayanan

2019

2019 IEEE Information Theory Workshop (ITW)

View full text Add to dashboard Cite

We show that polynomial codes (and some related codes) used for distributed matrix multiplication are interleaved Reed-Solomon codes and, hence, can be collaboratively decoded. We consider a fault tolerant setup where t worker nodes return erroneous values. For an additive random Gaussian error model, we show that for all t < N − K, errors can be corrected with probability 1. Further, numerical results show that in the presence of additive errors, when L Reed-Solomon codes are collaboratively decoded, the numerical stability in recovering the error locator polynomial improves with increasing L. Index TermsDistributed computation, collaborative decoding, polynomial codes I. INTRODUCTION AND MAIN RESULTWe consider the problem of computing A T B for two matrices A ∈ F s×r and B ∈ F s×r ′ (for an arbitrary field F) 1 in a distributed fashion with N worker nodes using a coded matrix multiplication scheme [1]-[11] To keep the presentation clear, we will focus on one class of codes, namely Polynomial codes, and explain our results in relation to the Polynomial codes [1]; notwithstanding, our results also apply to Entangled Polynomial codes [2] and PolyDot codes [3]. We assume that the matrices A and B are split into m subblocks and n subblocks, respectively. These subblocks are encoded using a Polynomial code [2]. Each worker node performs a matrix multiplication and returns a matrix with a total of L = rr ′ mn elements (from F) to the master node.Our main interest is in the fault-tolerant setup where some of the N worker nodes return erroneous values. We say that an error pattern of Hamming weight t has occurred if t worker nodes return matrices 1 Some results in this paper will apply to specific fields and this will be clarified later.

show abstract

Universally Decodable Matrices for Distributed Matrix-Vector Multiplication

Cited by 47 publications

References 17 publications

Numerically Stable Polynomially Coded Computing

Numerically Stable Polynomially Coded Computing

Hierarchical coded matrix multiplication

Collaborative Decoding of Polynomial Codes for Distributed Computation

Contact Info

Product

Resources

About