Speeding up distributed machine learning using codes

Abstract

Codes are widely used in many engineering applications to offer robustness against noise. In large-scale systems there are several types of noise that can affect the performance of distributed machine learning algorithms -straggler nodes, system failures, or communication bottlenecks -but there has been little interaction cutting across codes, machine learning, and distributed systems. In this work, we provide theoretical insights on how coded solutions can achieve significant gains compared to uncoded ones. We focus on two of the most basic building blocks of distributed learning algorithms: matrix multiplication and data shuffling. For matrix multiplication, we use codes to alleviate the effect of stragglers, and show that if the number of homogeneous workers is n, and the runtime of each subtask has an exponential tail, coded computation can speed up distributed matrix multiplication by a factor of log n. For data shuffling, we use codes to reduce communication bottlenecks, exploiting the excess in storage. We show that when a constant fraction α of the data matrix can be cached at each worker, and n is the number of workers, coded shuffling reduces the communication cost by a factor of (α + 1 n )γ(n) compared to uncoded shuffling, where γ(n) is the ratio of the cost of unicasting n messages to n users to multicasting a common message (of the same size) to n users. For instance, γ(n) n if multicasting a message to n users is as cheap as unicasting a message to one user. We also provide experiment results, corroborating our theoretical gains of the coded algorithms.

Untitled

“…There is also an emerging body of work on using replication or erasure coding to mitigate stragglers in linear computations, such as matrix-vector multiplication (Dutta et al 2016;Lee et al 2016;Mallick et al 2018) and matrix-matrix multiplication (Yang et al 2017;Yu et al 2017), and machine learning (Ferdinand and Draper 2016;Tandon et al 2017). Our work is for general (possibly nonlinear) computations for which coding techniques cannot be directly applied, and we have to resort to simpler task replication strategies.…”

Straggler-Aware Distributed Learning: Communication–Computation Latency Trade-Off

“…A wealth of straggler avoidance techniques have been proposed in recent years for DGD as well as other distributed computation tasks [ 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 ]. The common design notion behind all these schemes is the assignment of redundant computations/tasks to workers, such that faster workers can compensate for the stragglers.…”

Index Codes with Minimum Locality for Three Receiver Unicast Problems

“…The broadcast rate or the rate of an index code is the ratio of the code length to the length of each of the messages. The problem of designing index codes with smallest possible broadcast rate is significant because of its applications, such as multimedia content delivery [3], coded caching [4], distributed computation [5], and also because of its relation to network coding [6], [7] and coding for distributed storage [8], [9].…”