Improving Distributed Gradient Descent Using Reed-Solomon Codes

Abstract

Today's massively-sized datasets have made it necessary to often perform computations on them in a distributed manner. In principle, a computational task is divided into subtasks which are distributed over a cluster operated by a taskmaster. One issue faced in practice is the delay incurred due to the presence of slow machines, known as stragglers. Several schemes, including those based on replication, have been proposed in the literature to mitigate the effects of stragglers and more recently, those inspired by coding theory have begun to gain traction. In this work, we consider a distributed gradient descent setting suitable for a wide class of machine learning problems. We adapt the framework of Tandon et al. [18] and present a deterministic scheme that, for a prescribed per-machine computational effort, recovers the gradient from the least number of machines f theoretically permissible, via an O(f 2 ) decoding algorithm. We also provide a theoretical delay model which can be used to minimize the expected waiting time per computation by optimally choosing the parameters of the scheme. Finally, we supplement our theoretical findings with numerical results that demonstrate the efficacy of the method and its advantages over competing schemes.2. We provide an efficient online decoder, with time complexity O(f 2 ) for recovering the gradient from any f machines, which is faster than the best known method [18], O(f 3 ).3. We analyze the total computation time, and provide a method for finding the optimal coding parameters. We consider heavy-tailed delays, which have been widely observed in CPU job runtimes in practice [11,7,8].The rest of the paper is organized as follow. In Section 2, we describe the problem setup and explain the design objectives in detail. Section 3, provides the construction of our coding scheme, using the idea of balanced Reed-Solomon codes. Our efficient online decoder is presented in Section 4. We then characterize the total computation time, and describe the optimal choice of coding parameters, in Section 5. Finally, we provide our numerical results in Section 6, and conclude in Section 7.

Weighted Gradient Coding with Leverage Score Sampling

“…Each column of the encoding matrix B corresponds to a partition D i and is associated with a polynomial that evaluates to zero at the respective workers who have not been assigned that partition part. For more details the reader is referred to [2]. The matrixT = T · diag(w) is equal to T with its columns each scaled by the respective entry of w, thus T (1) = w. A direct consequence of this is that a T IB I = e T 1T =T (1) = w, which completes the proof.…”

“…Gradient coding requires the central server to receive the subtasks of a fixed fraction of any of the workers. We obtain an extension based on balanced Reed-Solomon codes [2,14], introducing weighted gradient coding, where the central server recovers a weighted sum of the partial gradients of the loss function. .…”

“…Let B and a I be an encoding matrix and decoding vector from[2], satisfying a T I B I = 1 1×k for any I. LetB := B · diag(w) for any w ∈ C 1×k . Then a T IB I = w. Proof.…”

LAGC: Lazily Aggregated Gradient Coding for Straggler-Tolerant and Communication-Efficient Distributed Learning

“…The PS then updates θ θ θ i+1 , as well as θ θ θ i 1 = θ θ θ i and θ θ θ i m = θ θ θ i−1 m for m = 2, 3. The next iteration i + 1 then continues with a check of condition (13) by the PS in the same way.…”

Hierarchical coded matrix multiplication

“…It was presented in part in ISIT'18 [1] [2], CWIT'19 [3], and ICML'19 [4]. computing gradients, thus accelerating the training of largescale machine learning applications [12], [13], [14], [15], [16]. While matrix multiplication and gradient descent are two types of coded computing problems that have been studied, others include coded convolution [17], coded approximate computing [18], sparse coded matrix multiplication [19], and heterogeneous coded computing [20].…”