Sharp Analysis for Nonconvex SGD Escaping from Saddle Points

Abstract

In this paper, we give a sharp analysis 1 for Stochastic Gradient Descent (SGD) and prove that SGD is able to efficiently escape from saddle points and find an ( , O( 0.5 ))-approximate secondorder stationary point in Õ( −3.5 ) stochastic gradient computations for generic nonconvex optimization problems, when the objective function satisfies gradient-Lipschitz, Hessian-Lipschitz, and dispersive noise assumptions. This result subverts the classical belief that SGD requires at least O( −4 ) stochastic gradient computations for obtaining an ( , O( 0.5 ))-approximate secondorder stationary point. Such SGD rate matches, up to a polylogarithmic factor of problemdependent parameters, the rate of most accelerated nonconvex stochastic optimization algorithms that adopt additional techniques, such as Nesterov's momentum acceleration, negative curvature search, as well as quadratic and cubic regularization tricks. Our novel analysis gives new insights into nonconvex SGD and can be potentially generalized to a broad class of stochastic optimization algorithms.

Escaping Saddle Points with Compressed SGD

“…When Assumption C holds, the communication can probably be improved by the factor of ε − 1 /4 using techniques from Fang et al [2019], which achieve Õ(ε −3.5 ) convergence rate under Assumption C outperforming Õ(ε −4 ) from Jin et al [2021] by the factor of ε − 1 /2 . When balancing the terms in Theorems 3.3 and 3.4, the communication improvement will be the square root of this value.…”

“…There are also a number of algorithms designed for finite sum setting where f (x) = n i=1 f i (x) [Reddi et al, 2017, Allen-Zhu and Li, 2018, Fang et al, 2018, or in case when only stochastic gradients are available [Tripuraneni et al, 2018, Jin et al, 2021, including variance reduction techniques [Allen-Zhu, 2018, Fang et al, 2018]. The sharpest rates in these settings have been obtained by Fang et al [2018], Zhou and Gu [2019] and Fang et al [2019].…”

“…Compared with the uncompressed case, the total communication improves by ε − 1 /4 when the stochastic gradient is Lipschitz and by √ dε − 3 /4 otherwise (the sharpest results for SGD are by Fang et al [2019] and Jin et al [2021] respectively). In Theorem 1.1, we heavily rely on the following property of RandomK: when its randomness (i.e.…”

“…We use the following standard [Jin et al, 2021, Fang et al, 2019, Allen-Zhu, 2018, Zhou et al, 2018 assumptions about the objective function f : Assumption A f is f max -bounded, L-smooth and has ρ-Lipschitz Hessian, i.e. for all x, y:…”

Escape saddle points by a simple gradient-descent based algorithm

“…• Can we achieve the polynomial speedup in log n for more advanced stochastic optimization algorithms with complexity Õ(poly(log n)/ 3.5 ) [2,3,9,28,30] or Õ(poly(log n)/ 3 ) [8, 33]?…”