Distributed Stochastic Gradient Tracking Algorithm With Variance Reduction for Non-Convex Optimization

Abstract

This paper considers a distributed stochastic nonconvex optimization problem, where the nodes in a network cooperatively minimize a sum of L-smooth local cost functions with sparse gradients. By adaptively adjusting the stepsizes according to the historical (possibly sparse) gradients, a distributed adaptive gradient algorithm is proposed, in which a gradient tracking estimator is used to handle the heterogeneity between different local cost functions. We establish an upper bound on the optimality gap, which indicates that our proposed algorithm can reach a first-order stationary solution dependent on the upper bound on the variance of the stochastic gradients. Finally, numerical examples are presented to illustrate the effectiveness of the algorithm.

Tackling Data Heterogeneity: A New Unified Framework for Decentralized SGD with Sample-induced Topology

“…Li et al (2021) proposed a similar algorithm with a nested loop structure for the sake of improving its overall complexity. Xin et al (2020) and Jiang et al (2022) consider a similar GT-VR framework and obtain a linear rate for strongly convex problems and O (1/k) rate for non-convex setting, respectively. Similar attempts have been recently made towards composite optimization problems (Ye et al, 2020).…”