Nested Distributed Gradient Methods with Adaptive Quantized Communication

Berahas, Albert S.; Iakovidou, Charikleia; Wei, Ermin

doi:10.48550/arxiv.1903.08149

Cited by 3 publications

(2 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We then generalize the results to the case where the number of communication steps varies at every iteration, and the quantization is adaptive-NEAR-DGD + +Q. For brevity we omit some of the proofs, and refer interested readers to [44]. We make the following assumptions that are standard in the distributed optimization literature [10], [11], [30].…”

Section: Convergence Analysismentioning

confidence: 99%

Nested Distributed Gradient Methods with Adaptive Quantized Communication

Berahas

Iakovidou

Wei

2019

2019 IEEE 58th Conference on Decision and Control (CDC)

Self Cite

View full text Add to dashboard Cite

In this paper, we consider minimizing a sum of local convex objective functions in a distributed setting, where communication can be costly. We propose and analyze a class of nested distributed gradient methods with adaptive quantized communication (NEAR-DGD+Q). We show the effect of performing multiple quantized communication steps on the rate of convergence and on the size of the neighborhood of convergence, and prove R-Linear convergence to the exact solution with increasing number of consensus steps and adaptive quantization. We test the performance of the method, as well as some practical variants, on quadratic functions, and show the effects of multiple quantized communication steps in terms of iterations/gradient evaluations, communication and cost.

show abstract

Section: Convergence Analysismentioning

confidence: 99%

Nested Distributed Gradient Methods with Adaptive Quantized Communication

Berahas

Iakovidou

Wei

2019

2019 IEEE 58th Conference on Decision and Control (CDC)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Reisizadeh et al (2018a) propose an algorithm that can still converge, though at a slower rate than the exact scheme. Another line of work proposed adaptive schemes (with increasing compression accuracy) that converge at the expense of higher communication cost (Carli et al, 2010a;Doan et al, 2018;Berahas et al, 2019). For deep learning applications, Tang et al (2018) proposed the DCD algorithm that converges at the same rate than the centralized baseline, O 1 / √ nT + n /(ρ 2 T ) , though only for quantization with quality δ = max 1 − Θ(ρ 2 ), 1 /4 , which may be limiting (not allowing the for the desired compression ratio) on many network topologies (e.g.…”

Section: Related Workmentioning

confidence: 99%

Decentralized Deep Learning with Arbitrary Communication Compression

Koloskova¹,

Lin²,

Stich³

et al. 2019

Preprint

View full text Add to dashboard Cite

Decentralized training of deep learning models is a key element for enabling data privacy and on-device learning over networks, as well as for efficient scaling to large compute clusters. As current approaches suffer from limited bandwidth of the network, we propose the use of communication compression in the decentralized training context. We show that Choco-SGD-recently introduced and analyzed for strongly-convex objectives only-converges under arbitrary high compression ratio on general non-convex functions at the rate O 1/ √ nT where T denotes the number of iterations and n the number of workers. The algorithm achieves linear speedup in the number of workers and supports higher compression than previous state-of-the art methods. We demonstrate the practical performance of the algorithm in two key scenarios: the training of deep learning models (i) over distributed user devices, connected by a social network and (ii) in a datacenter (outperforming all-reduce time-wise).

show abstract

FlexPD: A Flexible Framework Of First-Order Primal-Dual Algorithms for Distributed Optimization

Mansoori

Wei²

2019

Preprint

View full text Add to dashboard Cite

Nested Distributed Gradient Methods with Adaptive Quantized Communication

Cited by 3 publications

References 35 publications

Nested Distributed Gradient Methods with Adaptive Quantized Communication

Nested Distributed Gradient Methods with Adaptive Quantized Communication

Decentralized Deep Learning with Arbitrary Communication Compression

FlexPD: A Flexible Framework Of First-Order Primal-Dual Algorithms for Distributed Optimization

Contact Info

Product

Resources

About