DESTRESS: Computation-Optimal and Communication-Efficient Decentralized Nonconvex Finite-Sum Optimization

Abstract

Emerging applications in multi-agent environments such as internet-of-things, networked sensing, autonomous systems and federated learning, call for decentralized algorithms for finite-sum optimizations that are resource-efficient in terms of both computation and communication. In this paper, we consider the prototypical setting where the agents work collaboratively to minimize the sum of local loss functions by only communicating with their neighbors over a predetermined network topology. We develop a new algorithm, called DEcentralized STochastic REcurSive gradient methodS (DESTRESS ) for nonconvex finite-sum optimization, which matches the optimal incremental first-order oracle (IFO) complexity of centralized algorithms for finding first-order stationary points, while maintaining communication efficiency. Detailed theoretical and numerical comparisons corroborate that the resource efficiencies of DESTRESS improve upon prior decentralized algorithms over a wide range of parameter regimes. DE-STRESS leverages several key algorithm design ideas including stochastic recursive gradient updates with mini-batches for local computation, gradient tracking with extra mixing (i.e., multiple gossiping rounds) for per-iteration communication, together with careful choices of hyper-parameters and new analysis frameworks to provably achieve a desirable computation-communication trade-off.

BEER: Fast $O(1/T)$ Rate for Decentralized Nonconvex Optimization with Communication Compression

“…Among them, gradient tracking (Qu and Li, 2017;Di Lorenzo and Scutari, 2016;Nedic et al, 2017), which applies the idea of dynamic average consensus (Zhu and Martínez, 2010) to global gradient estimation, provides a systematic approach to reduce the variance and has been successfully applied to decentralize many algorithms with faster rates of convergence (Li et al, 2020a;Sun et al, 2019). For nonconvex problems, a small sample of gradient tracking aided algorithms include GT-SAGA (Xin et al, 2021), D-GET (Sun et al, 2020), GT-SARAH (Xin et al, 2020), and DESTRESS (Li et al, 2021a). Our BEER algorithm also leverages gradient tracking to eliminate the strong bounded gradient and bounded dissimilarity assumptions.…”

“…Note that Theorem 4.2 is a strict generalization of Theorem 4.1, and thus we will directly prove Theorem 4.2. This proof makes use of Lemma B.3 and Lemma B.4, by constructing some proper Lyapunov function and demonstrate its descending property using a linear system argument, which is also used in, e.g.,Li et al (2021a);Liao et al (2021).…”

Tackling Data Heterogeneity: A New Unified Framework for Decentralized SGD with Sample-induced Topology

“…For instance, Sun et al (2020) employ a scheme with both gradient-tracking and variancereduction to solve a smooth (probably non-convex) problem and show that it converges to a stationary point sublinearly. Li et al (2021) proposed a similar algorithm with a nested loop structure for the sake of improving its overall complexity. Xin et al (2020) and Jiang et al (2022) consider a similar GT-VR framework and obtain a linear rate for strongly convex problems and O (1/k) rate for non-convex setting, respectively.…”