A Unified Coding Framework for Distributed Computing with Straggling Servers

Li, Songze; Maddah-Ali, Mohammad Ali; Avestimehr, A. Salman

doi:10.1109/glocomw.2016.7848828

Cited by 179 publications

(214 citation statements)

References 13 publications

(40 reference statements)

Supporting

Mentioning

213

Contrasting

Order By: Relevance

“…The authors of [77] propose a new coded shuffling scheme based on pliable index coding. While most of the existing works focus on either coded computation or coded shuffling, one notable exception is [78]. In this work, the authors generalize the original coded MapReduce framework by introducing stragglers to the computation phases.…”

Section: B Data Shuffling and Communication Overheadsmentioning

confidence: 99%

Speeding up distributed machine learning using codes

Lee

Lam

Pedarsani

et al. 2016

2016 IEEE International Symposium on Information Theory (ISIT)

317

813

View full text Add to dashboard Cite

Codes are widely used in many engineering applications to offer robustness against noise. In large-scale systems there are several types of noise that can affect the performance of distributed machine learning algorithms -straggler nodes, system failures, or communication bottlenecks -but there has been little interaction cutting across codes, machine learning, and distributed systems. In this work, we provide theoretical insights on how coded solutions can achieve significant gains compared to uncoded ones. We focus on two of the most basic building blocks of distributed learning algorithms: matrix multiplication and data shuffling. For matrix multiplication, we use codes to alleviate the effect of stragglers, and show that if the number of homogeneous workers is n, and the runtime of each subtask has an exponential tail, coded computation can speed up distributed matrix multiplication by a factor of log n. For data shuffling, we use codes to reduce communication bottlenecks, exploiting the excess in storage. We show that when a constant fraction α of the data matrix can be cached at each worker, and n is the number of workers, coded shuffling reduces the communication cost by a factor of (α + 1 n )γ(n) compared to uncoded shuffling, where γ(n) is the ratio of the cost of unicasting n messages to n users to multicasting a common message (of the same size) to n users. For instance, γ(n) n if multicasting a message to n users is as cheap as unicasting a message to one user. We also provide experiment results, corroborating our theoretical gains of the coded algorithms.

show abstract

Section: B Data Shuffling and Communication Overheadsmentioning

confidence: 99%

Speeding up distributed machine learning using codes

Lee

Lam

Pedarsani

et al. 2016

2016 IEEE International Symposium on Information Theory (ISIT)

317

813

View full text Add to dashboard Cite

show abstract

“…Definition 5. The ensemble of N ×K generator matrices obtained by choosing the generator matrix G as in (9) where p i,j , q i,j are realizations of random variables P i,j , Q i,j such that {P 1,1 , . .…”

Section: A Encodingmentioning

confidence: 99%

Random Khatri-Rao-Product Codes for Numerically-Stable Distributed Matrix Multiplication

Subramaniam

Heidarzadeh

Narayanan

2019

2019 57th Annual Allerton Conference on Communication, Control, and Computing (Allerton)

View full text Add to dashboard Cite

We propose a class of codes called random Khatri-Rao-Product (RKRP) codes for distributed matrix multiplication in the presence of stragglers. The main advantage of the proposed codes is that decoding of RKRP codes is highly numerically stable in comparison to decoding of Polynomial codes [1] and decoding of the recently proposed OrthoPoly codes [2]. We show that RKRP codes are maximum distance separable with probability 1. The communication cost and encoding complexity for RKRP codes are identical to that of OrthoPoly codes and Polynomial codes and the average decoding complexity of RKRP codes is lower than that of OrthoPoly codes. Numerical results show that the average relative L 2 -norm of the reconstruction error for RKRP codes is substantially better than that of OrthoPoly codes.

show abstract

“…Following the results in [24], coded computing strategies have been proposed to efficiently deal with the stragglers for various computation tasks and network settings (see, e.g., [37]- [40]). In [41], we have superimposed the proposed CDC scheme on top of the MDS codes, developing a unified coding framework for distributed computing with straggling servers. This framework achieves a flexible tradeoff between computation latency in the Map phase and communication load in the Shuffle phase, which has the CDC scheme (or minimum bandwidth code) and the MDS code (or minimum latency code) as the two end points.…”

Section: Concluding Remarks and Future Directionsmentioning

confidence: 99%

Fundamental tradeoff between computation and communication in distributed computing

Maddah-Ali

Avestimehr

2016

2016 IEEE International Symposium on Information Theory (ISIT)

Self Cite

165

551

View full text Add to dashboard Cite

Abstract-How can we optimally trade extra computing power to reduce the communication load in distributed computing? We answer this question by characterizing a fundamental tradeoff between computation and communication in distributed computing, i.e., the two are inversely proportional to each other.More specifically, a general distributed computing framework, motivated by commonly used structures like MapReduce, is considered, where the overall computation is decomposed into computing a set of "Map" and "Reduce" functions distributedly across multiple computing nodes. A coded scheme, named "Coded Distributed Computing" (CDC), is proposed to demonstrate that increasing the computation load of the Map functions by a factor of r (i.e., evaluating each function at r carefully chosen nodes) can create novel coding opportunities that reduce the communication load by the same factor.An information-theoretic lower bound on the communication load is also provided, which matches the communication load achieved by the CDC scheme. As a result, the optimal computation-communication tradeoff in distributed computing is exactly characterized.Finally, the coding techniques of CDC is applied to the Hadoop TeraSort benchmark to develop a novel CodedTeraSort algorithm, which is empirically demonstrated to speed up the overall job execution by 1.97× -3.39×, for typical settings of interest.

show abstract

A Unified Coding Framework for Distributed Computing with Straggling Servers

Cited by 179 publications

References 13 publications

Speeding up distributed machine learning using codes

Speeding up distributed machine learning using codes

Random Khatri-Rao-Product Codes for Numerically-Stable Distributed Matrix Multiplication

Fundamental tradeoff between computation and communication in distributed computing

Contact Info

Product

Resources

About