Distributed Algorithms for Composite Optimization: Unified Framework and Convergence Analysis

Abstract

We study distributed composite optimization over networks: agents minimize a sum of smooth (strongly) convex functions-the agents' sum-utility-plus a nonsmooth (extendedvalued) convex one. We propose a general unified algorithmic framework for such a class of problems and provide a unified convergence analysis leveraging the theory of operator splitting. Distinguishing features of our scheme are: (i) When the agents' functions are strongly convex, the algorithm converges at a linear rate, whose dependence on the agents' functions and network topology is decoupled, matching the typical rates of centralized optimization; the rate expression improves on existing results; (ii) When the objective function is convex (but not strongly convex), similar separation as in (i) is established for the coefficient of the proved sublinear rate; (iii) The algorithm can adjust the ratio between the number of communications and computations to achieve a rate (in terms of computations) independent on the network connectivity; and (iv) A by-product of our analysis is a tuning recommendation for several existing (non accelerated) distributed algorithms yielding the fastest provably (worst-case) convergence rate. This is the first time that a general distributed algorithmic framework applicable to composite optimization enjoys all such properties.

Innovation Compression for Communication-efficient Distributed Optimization with Linear Convergence

“…To achieve a faster convergence rate, we are motivated by the state-of-the-art NIDS [7] that has a linear convergence rate O(max{ L µ , 1 ρ } log 1 ) to find an -optimal solution of (1), i.e., x i − x 2 ≤ , ∀i ∈ V [37]. NIDS can be written compactly as follows (6) where W = 1 2 (I + W ) and the first iteration is initialized as X 1 = X 0 − γ∇F (X 0 ).…”

A Hybrid Variance-Reduced Method for Decentralized Stochastic Non-Convex Optimization

“…Recent works [11][12][13][14] achieve robustness to heterogeneous environments by leveraging certain decentralized bias-correction techniques such as EXTRA (type) [15][16][17][18], gradient tracking [19][20][21][22][23][24][25], and primal-dual principles [16,[26][27][28]. Built on top of these bias-correction techniques, very recent works [29] and [30] propose D-GET and D-SPIDER-SFO respectively that further incorporate online SARAH/SPIDER-type variance reduction schemes [31][32][33] to achieve lower oracle complexities, when the SFO satisfies a mean-squared smoothness property.…”

Dual-Free Stochastic Decentralized Optimization with Variance Reduction

“…We would like to highlight the fact that the convergence theory of DVR decomposes nicely into several building blocks, and thus simple rates are obtained. This is not so usual for decentralized algorithms, for instance many follow-up papers were needed to obtain a tight convergence theory for EXTRA [Shi et al, 2015, Jakovetić, 2018, Xu et al, 2020, Li and Lin, 2020. We now discuss the convergence rate of DVR more in details.…”

“…Decentralized adaptations of gradient descent in the smooth and strongly convex setting include EXTRA [Shi et al, 2015], DIGing [Nedic et al, 2017] or NIDS [Li et al, 2019]. These algorithms have sparked a lot of interest, and the latest convergence results [Jakovetić, 2018, Xu et al, 2020, Li and Lin, 2020 show that EXTRA and NIDS require time O((κ b + γ −1 )(m + τ )) log(ε −1 )) to reach precision ε. A generic acceleration of EXTRA using Catalyst [Li and Lin, 2020] obtains the (batch) optimal O( √ κ b (1 + τ / √ γ) log(ε −1 )) rate up to log factors.…”