Accelerated Gradient Descent Escapes Saddle Points Faster than Gradient Descent

Abstract

Nesterov's accelerated gradient descent (AGD), an instance of the general family of "momentum methods," provably achieves faster convergence rate than gradient descent (GD) in the convex setting. However, whether these methods are superior to GD in the nonconvex setting remains open. This paper studies a simple variant of AGD, and shows that it escapes saddle points and finds a second-order stationary point in Õ(1/ǫ 7/4 ) iterations, faster than the Õ(1/ǫ 2 ) iterations required by GD. To the best of our knowledge, this is the first Hessian-free algorithm to find a second-order stationary point faster than GD, and also the first single-loop algorithm with a faster rate than GD even in the setting of finding a first-order stationary point. Our analysis is based on two key ideas: (1) the use of a simple Hamiltonian function, inspired by a continuous-time perspective, which AGD monotonically decreases per step even for nonconvex functions, and (2) a novel framework called improve or localize, which is useful for tracking the long-term behavior of gradient-based optimization algorithms. We believe that these techniques may deepen our understanding of both acceleration algorithms and nonconvex optimization. Guarantees Oracle AlgorithmIterations Simplicity First-order Stationary PointGradient GD [Nesterov, 1998] O(1/ǫ 2 ) Single-loop AGD [Ghadimi and Lan, 2016] O(1/ǫ 2 ) Single-loop Carmon et al. [2017] O 1/ǫ 7/4 Nested-loop Second-order Stationary Point Hessian -vector Carmon et al. [2016] O(1/ǫ 7/4 ) Nested-loop Agarwal et al. [2017] O(1/ǫ 7/4 ) Nested-loop Gradient Noisy GD [Ge et al., 2015] O(poly(d/ǫ)) Single-loop Perturbed GD [Jin et al., 2017] O(1/ǫ 2 ) Single-loop Perturbed AGD [This Work] O(1/ǫ 7/4 ) Single-loop Table 1: Complexity of finding stationary points. O(•) ignores polylog factors in d and ǫ.Convergence to first-order stationary points: Traditional analyses in this case assume only Lipschitz gradients (see Definition 1). Nesterov [1998] shows that GD finds an ǫ-first-order stationary point in O(1/ǫ 2 ) steps. Ghadimi and Lan [2016] guarantee that AGD also converges in O 1/ǫ 2 steps. Under the additional assumption of Lipschitz Hessians (see Definition 4), Carmon et al. [2017] develop a new algorithm that converges in O(1/ǫ 7/4 ) steps. Their algorithm is a nested-loop algorithm, where the outer loop adds a proximal term to reduce the nonconvex problem to a convex subproblem. A key novelty in their algorithm is the idea of "negative curvature exploitation," which inspired a similar step in our algorithm. In addition to the qualitative and quantitative differences between Carmon et al. [2017] and the current work, as summarized in Table 1, we note that while Carmon et al. [2017] analyze AGD applied to convex subproblems, we analyze AGD applied directly to nonconvex functions through a novel Hamiltonian framework.Convergence to second-order stationary points: All results in this setting assume Lipschitz conditions for both the gradient and Hessian. Classical approaches, such as cubic regularization [N...

Escape saddle points by a simple gradient-descent based algorithm

“…Our main contribution is a simple, single-loop, and robust gradient-based algorithm that can find an -approximate second-order stationary point of a smooth, Hessian Lipschitz function f : R n → R. Compared to previous works [3,24,29] exploiting the idea of gradient-based Hessian power method, our algorithm has a single-looped, simpler structure and better numerical stability. Compared to the previous state-of-the-art results with single-looped structures by [21] and [19,20] using Õ(log 6 n/ 1.75 ) or Õ(log 4 n/ 2 ) iterations, our algorithm achieves a polynomial speedup in log n: Theorem 1 (informal). Our single-looped algorithm finds an -approximate second-order stationary point in Õ(log n/ 1.75 ) iterations.…”

“…A seminal work along this line was by Ge et al [11], which found an -approximate second-order stationary point satisfying (1) using only gradients in O(poly(n, 1/ )) iterations. This is later improved to be almost dimension-free Õ(log 4 n/ 2 ) in the follow-up work [19], 2 and the perturbed accelerated gradient descent algorithm [21] based on Nesterov's accelerated gradient descent [26] takes Õ(log 6 n/ 1.75 ) iterations. However, these results still suffer from a significant overhead in terms of log n. On the other direction, Refs.…”

“…Technically, our work is inspired by the perturbed gradient descent (PGD) algorithm in [19,20] and the perturbed accelerated gradient descent (PAGD) algorithm in [21]. Specifically, PGD applies gradient descents iteratively until it reaches a point with small gradient, which can be a potential saddle point.…”

“…It is demonstrated that, with an appropriate choice of the perturbation radius, PGD can shake the point off from the neighborhood of the saddle point and converge to a second-order stationary point with high probability. The PAGD in [21] adopts a similar perturbation idea, but the GD is replaced by Nesterov's AGD [26].…”

“…See Proposition 3 and Proposition 5. After escaping the saddle point, similar to PGD and PAGD, we switch back to GD and AGD iterations, which are efficient to decrease the function value when the gradient is large [19,20,21].…”