GNSD: a Gradient-Tracking Based Nonconvex Stochastic Algorithm for Decentralized Optimization

Gt-Storm

“…• We conduct extensive experiments to examine the performance of our algorithm, including both a non-convex logistic regression model on the LibSVM datasets and convolutional neural network models on MNIST and CIFAR-10 datasets. Our experiments show that the our algorithm outperforms two state-of-the-art decentralized learning algorithms [19,20]. These experiments corroborate our theoretical results.…”

“…• Unlike existing approaches, our proposed GT-STORM algorithm adopts a new estimator, which is updated with a consensus mixing of the neighboring estimators of the last iteration, which helps improve the global gradient estimation. Our method achieves the nice features of previous works [6,9,20,33] while avoiding their pitfalls. To some extent, our GT-STORM algorithm can be viewed as an indirect way of integrating the stochastic gradient, variance reduction, and gradient tracking methods.…”

“…However, these methods require a full gradient evaluation per iteration, which yields $ (=n 2 ) sample complexity. To reduce the per-iteration sample complexity, stochastic gradients are adopted in the decentralized optimization (e.g., DSGD [13], PSGD [19], GNSD [20]). Due to the randomness in stochastic gradients, the convergence rate is reduced to $ (1/ p ) ).…”

“…To our knowledge, the decentralized gradient estimation and tracking (D-GET) algorithm in [32] is the only work for non-convex optimization. D-GET integrates the decentralized gradient tracking [20] and the SpiderBoost gradient estimator [36] to obtain $ (<= + < p =n 2 ) dataset-size-dependent sample complexity and $ (n 2 ) communication complexity. Recall that the sample and communication complexities of GT-STORM are $ (< 1/2 n 3 ) and $ (< 1/2 n 3 ), respectively.…”

“…Under some mild assumptions and parameter conditions, our algorithm enjoys an$ () 2/3 ) convergence rate. Note that this rate is much faster than the rate of $ () 1/2 ) for the classic decentralized stochastic algorithms, e.g., DSGD [13], PSGD [19] and GNSD [20]. Also, we show that to reach an n 2 -stationary solution, the total number of sample evaluations of our algorithm is$ (< 1/2 n 3 ) and the communication round is$ (< 1/2 n 3 ).…”