Robust Distributed Accelerated Stochastic Gradient Methods for Multi-Agent Networks

Abstract

We study distributed stochastic gradient (D-SG) method and its accelerated variant (D-ASG) for solving decentralized strongly convex stochastic optimization problems where the objective function is distributed over several computational units, lying on a fixed but arbitrary connected communication graph, subject to local communication constraints where noisy estimates of the gradients are available. We develop a framework which allows to choose the stepsize and the momentum parameters of these algorithms in a way to optimize performance by systematically trading off the bias, variance and dependence to network effects. When gradients do not contain noise, we also prove that D-ASG can achieve acceleration, in the sense that it requires O( √ κ log(1/ε)) gradient evaluations and O( √ κ log(1/ε)) communications to converge to the same fixed point with the non-accelerated variant where κ is the condition number and ε is the target accuracy. For quadratic functions, we also provide finer performance bounds that are tight with respect to bias and variance terms. Finally, we study a multistage version of D-ASG with parameters carefully varied over stages to ensure exact convergence to the optimal solution. It achieves optimal and accelerated O(−k/ √ κ) linear decay in the bias term as well as optimal O(σ 2 /k) in the variance 1

Optimal Gradient Tracking for Decentralized Optimization

“…and τ is chosen as a constant in (0, 1) such as τ = 1 2 . In light of the magnitudes mentioned in (12), inequalities ( 13) can be satisfied easily by choosing the parameters properly.…”

“…Since Φ k C is the weighted sum of consensus errors, inequality ( 14) also indicates that the weighted sum of consensus errors is a "Q-linear" sequence with "additional errors" in term of c 2 (γ, τ, p, q, η) ∇F X k − ∇F Q k 2 F . By the magnitudes of the parameters and stepsize mentioned in (12), we have…”

“…Remark 4. To see why the requirements (30a) and (30b) can be satisfied, we can substitute the magnitudes of the parameters and the stepsize in (12) and find that the magnitudes on both sides of the inequalities in (30a) and (30b) are actually all the same. But the power numbers of η or p on the LHS is higher than those on the RHS.…”

“…It was shown in [46] that to obtain ǫ-optimal solutions, the gradient computation complexity is lower bounded by O √ κ log 1 ǫ , and the communication complexity is lower bounded by O κ θ log 1 ǫ . To obtain better complexities, many accelerated decentralized gradient-type methods have been developed (e.g., [11,12,16,18,20,21,22,23,24,38,41,42,43,46,54,57,61,62]). There exist dual-based methods such as [46] that achieve optimal complexities.…”

Asymptotic Properties of $\mathcal{S}$-$\mathcal{AB}$ Method with Diminishing Stepsize

“…Distributed optimization problem has many applications in large-scale machine learning [1], wireless networks [2], parameter estimation [3], to name a few. Over the last decades, numerous algorithms for distributed optimization problem have been developed, such as (sub)gradient method [4][5][6][7][8][9][10], dual averaging method [11,12], primal dual method [13,14], gradient push method [15][16][17], gradient tracking method [18][19][20]. We refer to the survey [21] for the new development on distributed optimization.…”