Distributed Gradient Descent with Coded Partial Gradient Computations

Ozfatura, Emre; Ulukuş, Şennur; Gündüz, Deniz

doi:10.1109/icassp.2019.8683267

Cited by 43 publications

(28 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…in the communication load. As a future extension, we will design a partially recoverable GC scheme, similarly to [16], in order to provide further flexibility to the master.…”

mentioning

confidence: 99%

Gradient Coding with Clustering and Multi-Message Communication

Ozfatura

Gündüz

Ulukuş

2019

2019 IEEE Data Science Workshop (DSW)

View full text Add to dashboard Cite

Gradient descent (GD) methods are commonly employed in machine learning problems to optimize the parameters of the model in an iterative fashion. For problems with massive datasets, computations are distributed to many parallel computing servers (i.e., workers) to speed up GD iterations. While distributed computing can increase the computation speed significantly, the per-iteration completion time is limited by the slowest straggling workers. Coded distributed computing can mitigate straggling workers by introducing redundant computations; however, existing coded computing schemes are mainly designed against persistent stragglers, and partial computations at straggling workers are discarded, leading to wasted computational capacity. In this paper, we propose a novel gradient coding (GC) scheme which allows multiple coded computations to be conveyed from each worker to the master per iteration. We numerically show that the proposed GC with multi-message communication (MMC) together with clustering provides significant improvements in the average completion time (of each iteration), with minimal or no increase in the communication load.

show abstract

“…in the communication load. As a future extension, we will design a partially recoverable GC scheme, similarly to [16], in order to provide further flexibility to the master.…”

mentioning

confidence: 99%

Gradient Coding with Clustering and Multi-Message Communication

Ozfatura

Gündüz

Ulukuş

2019

2019 IEEE Data Science Workshop (DSW)

View full text Add to dashboard Cite

show abstract

“…As highlighted in [14] and [16], unlike most coded GD approaches, uncoded computations also allow implementing DSGD, which, in turn, reduces either the computation load of the workers, or the communication load. A coded DSGD algorithm is also proposed in [17].…”

Section: A Prior Workmentioning

confidence: 99%

Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air

Amiri

Gündüz

2019

2019 IEEE International Symposium on Information Theory (ISIT)

189

347

View full text Add to dashboard Cite

We study collaborative machine learning (ML) at the wireless edge, where power and bandwidth-limited wireless devices with local datasets carry out distributed stochastic gradient descent (DSGD) with the help of a remote parameter server (PS). Standard approaches assume separate computation and communication, where local gradient estimates are compressed and communicated to the PS over orthogonal links. Following this digital approach, we introduce D-DSGD, in which the wireless terminals, referred to as the workers, employ gradient quantization and error accumulation, and transmit their gradient estimates to the PS over the underlying wireless multiple access channel (MAC).We then introduce an analog scheme, called A-DSGD, which exploits the additive nature of the wireless MAC for over-the-air gradient computation. In A-DSGD, the workers first sparsify their gradient estimates, and then project them to a lower dimensional space imposed by the available channel bandwidth. These projections are transmitted directly over the MAC without employing any digital code. Numerical results show that A-DSGD converges much faster than D-DSGD thanks to its more efficient use of the limited bandwidth and the natural alignment of the gradient estimates over the channel. The improvement is particularly compelling at low power and low bandwidth regimes. We also observe that the performance of A-DSGD improves with the number of workers (keeping the total size of the dataset constant), while D-DSGD deteriorates, limiting the ability of the latter in harnessing the computation power of edge devices. The lack of quantization and channel encoding/decoding in A-DSGD further speeds up communication, making it very attractive for low-latency ML applications at the wireless network edge. 1 |Bm,t| u n ∈Bm,t ∇f (θ t , u n ) is the stochastic gradient of the current model computed at worker m, m ∈ [M ], using the

show abstract

“…Edge Computing: In the online phase, each EN k computes inner products between all users' data received in the uplink and the available coded model rows in set C k . As in [8], the order in which such computations are carried out is specified by vector…”

Section: System Model and Performance Criteria A System Modelmentioning

confidence: 99%

On Model Coding for Distributed Inference and Transmission in Mobile Edge Computing Systems

Zhang

Simeone

2019

IEEE Commun. Lett.

View full text Add to dashboard Cite

Consider a mobile edge computing system in which users wish to obtain the result of a linear inference operation on locally measured input data. Unlike the offloaded input data, the model weight matrix is distributed across wireless Edge Nodes (ENs). ENs have non-deterministic computing times, and they can transmit any shared computed output back to the users cooperatively. This letter investigates the potential advantages obtained by coding model information prior to ENs' storage. Through an information-theoretic analysis, it is concluded that, while generally limiting cooperation opportunities, coding is instrumental in reducing the overall computation-pluscommunication latency.

show abstract

Distributed Gradient Descent with Coded Partial Gradient Computations

Cited by 43 publications

References 16 publications

Gradient Coding with Clustering and Multi-Message Communication

Gradient Coding with Clustering and Multi-Message Communication

Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air

On Model Coding for Distributed Inference and Transmission in Mobile Edge Computing Systems

Contact Info

Product

Resources

About