CodedSketch: A Coding Scheme for Distributed Computation of Approximated Matrix Multiplication

Jahani-Nezhad, Tayyebeh; Maddah-Ali, Mohammad Ali

doi:10.48550/arxiv.1812.10460

Cited by 6 publications

(10 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Remark 9: An interpolation point x j is called unattainable if the interpolation condition is not satisfied, i.e., p(xj ) (10). Occurring unattainable points is one of the major flaws of traditional rational interpolants which is not occurred in Berrut's rational interpolant.…”

Section: Preliminariesmentioning

confidence: 99%

“…The general version of entangled polynomial codes is proposed in [9] to multiply more than two matrices. In [10], CodedSketch as a straggler-resistant coded scheme is introduced to compute the approximation of matrix multiplication where the exact result of the multiplication is not required. Lagrange codes [11] provide a novel strategy to compute an arbitrary polynomial function, without waiting for stragglers, and communication across worker nodes.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Berrut Approximated Coded Computing: Straggler Resistance Beyond Polynomial Computing

Jahani-Nezhad¹,

Maddah-Ali²

2020

Preprint

Self Cite

View full text Add to dashboard Cite

One of the major challenges in using distributed learning to train complicated models with large data sets is to deal with stragglers effect. As a solution, coded computation has been recently proposed to efficiently add redundancy to the computation tasks. In this technique, coding is used across data sets, and computation is done over coded data, such that the results of an arbitrary subset of worker nodes with a certain size are enough to recover the final results.The major challenges with those approaches are (1) they are limited to polynomial function computations, (2) the size of the subset of servers that we need to wait for grows with the multiplication of the size of the data set and the model complexity (the degree of the polynomial), which can be prohibitively large, (3) they are not numerically stable for computation over real numbers. In this paper, we propose Berrut Approximated Coded Computing (BACC), as an alternative approach, which is not limited to polynomial function computation. In addition, the master node can approximately calculate the final results, using the outcomes of any arbitrary subset of available worker nodes.The approximation approach is proven to be numerically stable with low computational complexity. In addition, the accuracy of the approximation is established theoretically and verified by simulation results in different settings such as distributed learning problems. In particular, BACC is used to train a deep neural network on a cluster of servers, which outperforms repetitive computation (repetition coding) in terms of the rate of convergence. I. INTRODUCTIONDistributed machine learning is known as an inevitable solution to overcome the challenge of training complicated models such as deep learning with large data sets [1]- [3]. In this solution, the data set or the model parameters are distributed among several servers and the tasks of training/evaluating are performed distributedly by those servers, in coordination with each other. In one scenario, for example, the parameters are maintained in a master node, while the data set is shared among some worker nodes. The worker nodes process data set locally and send the results to the master node to update the parameters. Distributed machine learning raises a list of challenges related to convergence rate, communication load, privacy, existence of faulty nodes, etc. One of the major challenges here is dealing with stragglers, or slow servers. Indeed, in those systems, the speed of computing is dominated

show abstract

Section: Preliminariesmentioning

confidence: 99%

mentioning

confidence: 99%

Berrut Approximated Coded Computing: Straggler Resistance Beyond Polynomial Computing

Jahani-Nezhad¹,

Maddah-Ali²

2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Recently, there has been extensive research activities to leverage coding schemes in order to boost the performance of distributed computing systems [6]- [18]. However, most of the work in the literature focus on the application of maximum distance separable (MDS) codes.…”

Section: Introductionmentioning

confidence: 99%

Coded Computing via Binary Linear Codes: Designs and Performance Limits

Soleymani¹,

Jamali²,

Mahdavifar³

2021

Preprint

View full text Add to dashboard Cite

We consider the problem of coded distributed computing where a large linear computational job, such as a matrix multiplication, is divided into k smaller tasks, encoded using an (n, k) linear code, and performed over n distributed nodes. The goal is to reduce the average execution time of the computational job. We provide a connection between the problem of characterizing the average execution time of a coded distributed computing system and the problem of analyzing the error probability of codes of length n used over erasure channels. Accordingly, we present closed-form expressions for the execution time using binary random linear codes and the best execution time any linear-coded distributed computing system can achieve. It is also shown that there exist good binary linear codes that not only attain (asymptotically) the best performance that any linear code (not necessarily binary) can achieve but also are numerically stable against the inevitable rounding errors in practice. We then develop a low-complexity algorithm for decoding Reed-Muller (RM) codes over erasure channels. Our decoder only involves additions and subtractions and enables coded computation over real-valued data. Extensive numerical analysis of the fundamental results as well as RM-and polarcoded computing schemes demonstrate the excellence of the RMcoded computation in achieving close-to-optimal performance while having a low-complexity decoding and explicit construction. The proposed framework in this paper enables efficient designs of distributed computing systems given the rich literature in the channel coding theory.

show abstract

“…On seemly irrelevant area, extensive efforts have been dedicated to using coding theory to improve the efficiency of distributed computing, mainly to cope with the stragglers [10]- [20]. The core idea is based on partitioning each input data into some smaller inputs and then encoding smaller inputs.…”

Section: Introductionmentioning

confidence: 99%

“…In particular, in [13], the code is designed such that the results of different workers form a maximum separable code (MDS), meaning that the final result can be recovered from any subset of servers with the minimum size. That approach has been extended to general matrix partitioning in [14], [15], and to the cases where only an approximate result of the matrix multiplication is needed [17], [20].…”

Section: Introductionmentioning

confidence: 99%

Limited-Sharing Multi-Party Computation for Massive Matrix Operations

Nodehi

Maddah-Ali

2018

2018 IEEE International Symposium on Information Theory (ISIT)

Self Cite

View full text Add to dashboard Cite

In this paper, we consider a secure multi-party computation problem (MPC), where the goal is to offload the computation of an arbitrary polynomial function of some massive private matrices to a network of workers. The workers are not reliable, some may collude to gain information about the input data. The system is initialized by sharing a (randomized) function of each input matrix to each server. Since the input matrices are massive, the size of each share is assumed to be at most 1/k fraction of the input matrix, for some k ∈ N. The objective is to minimize the number of workers needed to perform the computation task correctly, such that even if an arbitrary subset of t − 1 workers, for some t ∈ N, collude, they cannot gain any information about the input matrices. We propose a sharing scheme, called polynomial sharing, and show that it admits basic operations such as adding and multiplication of matrices, and transposing a matrix. By concatenating the procedures for basic operations, we show that any polynomial function of the input matrices can be calculated, subject to the problem constraints. We show that the proposed scheme can offer order-wise gain, in terms of number of workers needed, compared to the approaches formed by concatenation of job splitting and conventional MPC approaches.

show abstract

CodedSketch: A Coding Scheme for Distributed Computation of Approximated Matrix Multiplication

Cited by 6 publications

References 0 publications

Berrut Approximated Coded Computing: Straggler Resistance Beyond Polynomial Computing

Berrut Approximated Coded Computing: Straggler Resistance Beyond Polynomial Computing

Coded Computing via Binary Linear Codes: Designs and Performance Limits

Limited-Sharing Multi-Party Computation for Massive Matrix Operations

Contact Info

Product

Resources

About