Information Theoretic Limits of Data Shuffling for Distributed Learning

Attia, Mohamed Adel; Tandon, Ravi

doi:10.1109/glocom.2016.7841903

Cited by 38 publications

(34 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [75], the authors study the information-theoretic limits of the coded shuffling problem. More specifically, the authors completely characterize the fundamental limits for the case of 2 workers and the case of 3 workers.…”

Section: B Data Shuffling and Communication Overheadsmentioning

confidence: 99%

Speeding up distributed machine learning using codes

Lee

Lam

Pedarsani

et al. 2016

2016 IEEE International Symposium on Information Theory (ISIT)

317

813

View full text Add to dashboard Cite

Codes are widely used in many engineering applications to offer robustness against noise. In large-scale systems there are several types of noise that can affect the performance of distributed machine learning algorithms -straggler nodes, system failures, or communication bottlenecks -but there has been little interaction cutting across codes, machine learning, and distributed systems. In this work, we provide theoretical insights on how coded solutions can achieve significant gains compared to uncoded ones. We focus on two of the most basic building blocks of distributed learning algorithms: matrix multiplication and data shuffling. For matrix multiplication, we use codes to alleviate the effect of stragglers, and show that if the number of homogeneous workers is n, and the runtime of each subtask has an exponential tail, coded computation can speed up distributed matrix multiplication by a factor of log n. For data shuffling, we use codes to reduce communication bottlenecks, exploiting the excess in storage. We show that when a constant fraction α of the data matrix can be cached at each worker, and n is the number of workers, coded shuffling reduces the communication cost by a factor of (α + 1 n )γ(n) compared to uncoded shuffling, where γ(n) is the ratio of the cost of unicasting n messages to n users to multicasting a common message (of the same size) to n users. For instance, γ(n) n if multicasting a message to n users is as cheap as unicasting a message to one user. We also provide experiment results, corroborating our theoretical gains of the coded algorithms.

show abstract

Section: B Data Shuffling and Communication Overheadsmentioning

confidence: 99%

Speeding up distributed machine learning using codes

Lee

Lam

Pedarsani

et al. 2016

2016 IEEE International Symposium on Information Theory (ISIT)

317

813

View full text Add to dashboard Cite

show abstract

“…In other words, τ * is the time for which there are exactly r inner products -on average -aggregated at the master node, when the workers are loaded according to the loading obtained in (13). Using (13), (14) and (16), we find that…”

Section: B Solving the Alternate Formulationmentioning

confidence: 99%

Coded computation over heterogeneous clusters

Reisizadeh

Prakash

Pedarsani

et al. 2017

2017 IEEE International Symposium on Information Theory (ISIT)

108

156

View full text Add to dashboard Cite

In large-scale distributed computing clusters, such as Amazon EC2, there are several types of "system noise" that can result in major degradation of performance: system failures, bottlenecks due to limited communication bandwidth, latency due to straggler nodes, etc. On the other hand, these systems enjoy abundance of redundancy -a vast number of computing nodes and large storage capacity. There have been recent results that demonstrate the impact of coding for efficient utilization of computation and storage redundancy to alleviate the effect of stragglers and communication bottlenecks in homogeneous clusters. In this paper, we focus on general heterogeneous distributed computing clusters consisting of a variety of computing machines with different capabilities. We propose a coding framework for speeding up distributed computing in heterogeneous clusters by trading redundancy for reducing the latency of computation. In particular, we propose Heterogeneous Coded Matrix Multiplication (HCMM) algorithm for performing distributed matrix multiplication over heterogeneous clusters that is provably asymptotically optimal for a broad class of processing time distributions. Moreover, we show that HCMM is unboundedly faster than uncoded schemes that partition the total work load among the workers. To demonstrate how the proposed HCMM scheme can be applied in practice, we provide numerical results demonstrating significant speedups of up to 90% and 35% for HCMM in comparison to the "uncoded" and "coded homogeneous" schemes, respectively. Furthermore, we carry out real experiments over Amazon EC2 clusters that corroborate our numerical studies, where HCMM is found to be up to 17% faster than the uncoded scheme. Additionally, our observation is that machines rarely become stragglers and when they do, they continue to exhibit slower performance for sometime. In our worst case experiments with artificial stragglers, HCMM provides speedups of up to 12× over the uncoded scheme. Furthermore, we provide a generalization of the problem of optimal load allocation for heterogeneous clusters to scenarios with budget constraints and develop a heuristic algorithm for efficient load allocation. In the end, we discuss about the decoding complexity and describe how LDPC codes can be combined with HCMM in order to control the complexity of decoding as the problem size increases.

show abstract

“…where K|Q and each node is supposed to compute Q K functions. In fact, in this setup, the K in definitions (21) and (22) will be replaced by Q. Achievability proofs can be shown by executing Q K times the distributed computing schemes as explained in Section IV and V. The converse can be derived by adjusting the definitions in (119), (121), (194), and following the same steps in Appendix B.…”

Section: Definition 4 (Fundamental Scc Regionmentioning

confidence: 99%

Storage, Computation, and Communication: A Fundamental Tradeoff in Distributed Computing

Yan

Yang

Wigger

2018

2018 IEEE Information Theory Workshop (ITW)

View full text Add to dashboard Cite

Distributed computing has become one of the most important frameworks in dealing with large computation tasks. In this paper, we investigate a MapReduce like distributed computing system. Our main contribution is the characterization of the optimal tradeoff between storage space, computation load, and communication load. To this end, we derive an information-theoretical converse and show that time-and memory-sharing between the operating points achieved by the modified coded distributed computing (M-CDC) scheme proposed by Ezzeldin et al. attains all the points on this tradeoff. Our result thus extends the result by Li et al. on the optimum tradeoff between storage and communication to account also for the computation load. We further show how to obtain a distributed computing scheme from any placement delivery array (PDA) whose ordinary symbols occur at least twice. Previously proposed PDAs to solve the subpacketization problem in coded caching by Yan et al. and Shangguan et al. allow us then to derive optimal distributed computing schemes that require only a small number of files, and thus have reduced complexity.Recently, Li et al.[3] proposed a scheme, termed coded distributed computing (CDC), that in the map phase stores files multiple times across users so as to enable multicast opportunities for the shuffle phase. This approach can significantly reduce the communication load over traditional uncoded schemes as illustrated in Fig. 2, and was proved in [3] to have the smallest communication load among all the distributed computing schemes with same total storage requirements. It is worth mentioning that Li et al. in [3] used the term computation-communication tradeoff, because they assumed that each node calculates all the IVAs that can be obtained from the data stored at that node, irrespective of whether these IVAs are used in the sequel or not. In this sense, the total number of calculated IVAs is actually a measure of the total storage space consumed across the nodes. This is why we would rather refer to it as the storage-communication tradeoff. In this context, it is natural to investigate a more general framework, where each node is allowed to choose for each IVA that it can calculate from its locally stored data, whether or not to perform this calculation. The number of IVAs effectively calculated at all the nodes, normalized by the total number of IVAs, is then used to measure the real computation load. In this sense, we extend the storage-communication tradeoff in [3] to a storage-computation-communication tradeoff. That means, our goal is to identify all storage-computation-communication (SCC) triples, for which the task of computing the K output functions in (1) can be accomplished.August 28, 2018 DRAFT Fig. 2: Comparison of CDC scheme and uncoded scheme for a task of computing K = 10 output functions.In this work, we fully characterize all feasible SCC triples, and as a consequence, the pareto-optimal tradeoff surface. We show that any feasible triple on this optimal tradeoff surface can be attai...

show abstract

Information Theoretic Limits of Data Shuffling for Distributed Learning

Cited by 38 publications

References 4 publications

Speeding up distributed machine learning using codes

Speeding up distributed machine learning using codes

Coded computation over heterogeneous clusters

Storage, Computation, and Communication: A Fundamental Tradeoff in Distributed Computing

Contact Info

Product

Resources

About