Coded MapReduce

Li, Songze; Maddah-Ali, Mohammad Ali; Avestimehr, A. Salman

doi:10.1109/allerton.2015.7447112

Cited by 144 publications

(142 citation statements)

References 20 publications

(75 reference statements)

Supporting

Mentioning

141

Contrasting

Order By: Relevance

“…A preliminary part of this result, in particular the achievability for the special case of s = 1, or the achievable scheme of Theorem 1 was presented in [1]. We note that when s = 1, Theorem 2 provides the same result as in Theorem 1, i.e., L * (r, 1) = Remark 8.…”

Section: Problem Formulationmentioning

confidence: 55%

“…Here we do not impose any constraint on how the Map and Reduce functions are chosen (for example, they can be arbitrary linear or nonlinear functions). 1 When mapping a file, we compute Q intermediate values in parallel, one for each of the Q output functions. The main reason to do this is that parallel processing can be efficiently performed for applications that fit into the MapReduce framework.…”

Section: Problem Formulationmentioning

confidence: 99%

See 1 more Smart Citation

Fundamental tradeoff between computation and communication in distributed computing

Maddah-Ali

Avestimehr

2016

2016 IEEE International Symposium on Information Theory (ISIT)

Self Cite

165

551

View full text Add to dashboard Cite

Abstract-How can we optimally trade extra computing power to reduce the communication load in distributed computing? We answer this question by characterizing a fundamental tradeoff between computation and communication in distributed computing, i.e., the two are inversely proportional to each other.More specifically, a general distributed computing framework, motivated by commonly used structures like MapReduce, is considered, where the overall computation is decomposed into computing a set of "Map" and "Reduce" functions distributedly across multiple computing nodes. A coded scheme, named "Coded Distributed Computing" (CDC), is proposed to demonstrate that increasing the computation load of the Map functions by a factor of r (i.e., evaluating each function at r carefully chosen nodes) can create novel coding opportunities that reduce the communication load by the same factor.An information-theoretic lower bound on the communication load is also provided, which matches the communication load achieved by the CDC scheme. As a result, the optimal computation-communication tradeoff in distributed computing is exactly characterized.Finally, the coding techniques of CDC is applied to the Hadoop TeraSort benchmark to develop a novel CodedTeraSort algorithm, which is empirically demonstrated to speed up the overall job execution by 1.97× -3.39×, for typical settings of interest.

show abstract

Section: Problem Formulationmentioning

confidence: 55%

Section: Problem Formulationmentioning

confidence: 99%

Fundamental tradeoff between computation and communication in distributed computing

Maddah-Ali

Avestimehr

2016

2016 IEEE International Symposium on Information Theory (ISIT)

Self Cite

165

551

View full text Add to dashboard Cite

show abstract

“…Coded caching has been studied in many scenarios such as decentralized coded caching [68], online coded caching [69], hierarchical coded caching for wireless communication [70], and device-to-device coded caching [71]. Recently, the authors in [72] proposed coded MapReduce that reduces the communication cost in the process of transferring the results of mappers to reducers.…”

Section: B Data Shuffling and Communication Overheadsmentioning

confidence: 99%

Speeding up distributed machine learning using codes

Lee

Lam

Pedarsani

et al. 2016

2016 IEEE International Symposium on Information Theory (ISIT)

317

813

View full text Add to dashboard Cite

Codes are widely used in many engineering applications to offer robustness against noise. In large-scale systems there are several types of noise that can affect the performance of distributed machine learning algorithms -straggler nodes, system failures, or communication bottlenecks -but there has been little interaction cutting across codes, machine learning, and distributed systems. In this work, we provide theoretical insights on how coded solutions can achieve significant gains compared to uncoded ones. We focus on two of the most basic building blocks of distributed learning algorithms: matrix multiplication and data shuffling. For matrix multiplication, we use codes to alleviate the effect of stragglers, and show that if the number of homogeneous workers is n, and the runtime of each subtask has an exponential tail, coded computation can speed up distributed matrix multiplication by a factor of log n. For data shuffling, we use codes to reduce communication bottlenecks, exploiting the excess in storage. We show that when a constant fraction α of the data matrix can be cached at each worker, and n is the number of workers, coded shuffling reduces the communication cost by a factor of (α + 1 n )γ(n) compared to uncoded shuffling, where γ(n) is the ratio of the cost of unicasting n messages to n users to multicasting a common message (of the same size) to n users. For instance, γ(n) n if multicasting a message to n users is as cheap as unicasting a message to one user. We also provide experiment results, corroborating our theoretical gains of the coded algorithms.

show abstract

“…The first coding concept introduced in [6]- [8] enables an inverse-linear tradeoff between computation load and communication load in distributed computing. This result implies that increasing the computation load by a factor of r (i.e., evaluating each computation at r carefully chosen nodes) can create novel coding opportunities that reduce the required communication load for computing by the same factor r. Hence, these codes can be utilized to pool the underutilized computing resources at network edge to slash the communication load of Fog computing [9].…”

Section: Introductionmentioning

confidence: 99%

“…Using Lemma 3, the minimum and maximum induced costs for the task of computing r = 100 equations are C m = 314.6 and C M = 2516.8 respectively. It takes 15 iterations for the proposed heuristic search algorithm to arrive at the tuple (n 1 , n 2 , n 3 ) = (10,6,0). This corresponds to the expected cost 486.2 and the expected time E[T HCMM ] = 14.3.…”

mentioning

confidence: 99%

Coded computation over heterogeneous clusters

Reisizadeh

Prakash

Pedarsani

et al. 2017

2017 IEEE International Symposium on Information Theory (ISIT)

108

156

View full text Add to dashboard Cite

In large-scale distributed computing clusters, such as Amazon EC2, there are several types of "system noise" that can result in major degradation of performance: system failures, bottlenecks due to limited communication bandwidth, latency due to straggler nodes, etc. On the other hand, these systems enjoy abundance of redundancy -a vast number of computing nodes and large storage capacity. There have been recent results that demonstrate the impact of coding for efficient utilization of computation and storage redundancy to alleviate the effect of stragglers and communication bottlenecks in homogeneous clusters. In this paper, we focus on general heterogeneous distributed computing clusters consisting of a variety of computing machines with different capabilities. We propose a coding framework for speeding up distributed computing in heterogeneous clusters by trading redundancy for reducing the latency of computation. In particular, we propose Heterogeneous Coded Matrix Multiplication (HCMM) algorithm for performing distributed matrix multiplication over heterogeneous clusters that is provably asymptotically optimal for a broad class of processing time distributions. Moreover, we show that HCMM is unboundedly faster than uncoded schemes that partition the total work load among the workers. To demonstrate how the proposed HCMM scheme can be applied in practice, we provide numerical results demonstrating significant speedups of up to 90% and 35% for HCMM in comparison to the "uncoded" and "coded homogeneous" schemes, respectively. Furthermore, we carry out real experiments over Amazon EC2 clusters that corroborate our numerical studies, where HCMM is found to be up to 17% faster than the uncoded scheme. Additionally, our observation is that machines rarely become stragglers and when they do, they continue to exhibit slower performance for sometime. In our worst case experiments with artificial stragglers, HCMM provides speedups of up to 12× over the uncoded scheme. Furthermore, we provide a generalization of the problem of optimal load allocation for heterogeneous clusters to scenarios with budget constraints and develop a heuristic algorithm for efficient load allocation. In the end, we discuss about the decoding complexity and describe how LDPC codes can be combined with HCMM in order to control the complexity of decoding as the problem size increases.

show abstract

Coded MapReduce

Cited by 144 publications

References 20 publications

Fundamental tradeoff between computation and communication in distributed computing

Fundamental tradeoff between computation and communication in distributed computing

Speeding up distributed machine learning using codes

Coded computation over heterogeneous clusters

Contact Info

Product

Resources

About