Communication Compression for Distributed Nonconvex Optimization

Abstract

This paper considers decentralized nonconvex optimization with the cost functions being distributed over agents. Noting that information compression is a key tool to reduce the heavy communication load for decentralized algorithms as agents iteratively communicate with neighbors, we propose three decentralized primal-dual algorithms with compressed communication. The first two algorithms are applicable to a general class of compressors with bounded relative compression error and the third algorithm is suitable for two general classes of compressors with bounded absolute compression error.We show that the proposed decentralized algorithms with compressed communication have comparable convergence properties as state-of-the-art algorithms without communication compression. Specifically, we show that they can find first-order stationary points with sublinear convergence rate O(1/T ) when each local cost function is smooth, where T is the total number of iterations, and find global optima with linear convergence rate under an additional condition that the global cost function satisfies the Polyak-Łojasiewicz condition. Numerical simulations are provided to illustrate the effectiveness of the theoretical results.

Numerical Methods for Distributed Stochastic Compositional Optimization Problems with Aggregative Structure

“…In recent years, various techniques have been developed to reduce communication costs [27,31]. They are extensively incorporated into centralized optimization methods [1,23,30] and decentralized methods [8,9,18,20,33]. This motivates us to provide an extension of D-ASCGD by combining it with communication compressed method, which reads as follows.…”

“…Similar to (52), 24ȓ 2 (1+αyr 2 ψ 2 ) , (33) holds. Inequality (34) could be obtained by the similar analysis of (33).…”

“…For vector x ∈ R d or z ∈ R p , the class of compressors satisfying (28) or ( 29) is broad, such as random quantization [23,26], sparsification [15,19], the norm-sign compressor [18,33]. For matrix y ∈ R d×p , we may construct the dp-length vector by stacking up the columns of y and then implement predetermined compressors to compress the new constructed vector.…”

CEDAS: A Compressed Decentralized Stochastic Gradient Method with Improved Convergence

“…(2) primal-dual like methods [22,17,15,14,54,51]; (3) gradient tracking based algorithms [20,55,37,50,47].…”

“…The paper [22] equipped NIDS [18] with communication compression and demonstrated its linear convergence rate when the objective functions are smooth and strongly convex. More recently, the papers [20,37] considered a class of linearly convergent decentralized gradient tracking method with communication compression that applies to general directed graphs; see also [54,51] for more related works.…”