On Nonconvex Optimization for Machine Learning: Gradients, Stochasticity, and Saddle Points

Abstract

Gradient descent (GD) and stochastic gradient descent (SGD) are the workhorses of large-scale machine learning. While classical theory focused on analyzing the performance of these methods in convex optimization problems, the most notable successes in machine learning have involved nonconvex optimization, and a gap has arisen between theory and practice. Indeed, traditional analyses of GD and SGD show that both algorithms converge to stationary points efficiently. But these analyses do not take into account the possibility of converging to saddle points. More recent theory has shown that GD and SGD can avoid saddle points, but the dependence on dimension in these analyses is polynomial. For modern machine learning, where the dimension can be in the millions, such dependence would be catastrophic. We analyze perturbed versions of GD and SGD and show that they are truly efficient-their dimension dependence is only polylogarithmic. Indeed, these algorithms converge to second-order stationary points in essentially the same time as they take to converge to classical first-order stationary points.

Escape saddle points by a simple gradient-descent based algorithm

“…Our main contribution is a simple, single-loop, and robust gradient-based algorithm that can find an -approximate second-order stationary point of a smooth, Hessian Lipschitz function f : R n → R. Compared to previous works [3,24,29] exploiting the idea of gradient-based Hessian power method, our algorithm has a single-looped, simpler structure and better numerical stability. Compared to the previous state-of-the-art results with single-looped structures by [21] and [19,20] using Õ(log 6 n/ 1.75 ) or Õ(log 4 n/ 2 ) iterations, our algorithm achieves a polynomial speedup in log n: Theorem 1 (informal). Our single-looped algorithm finds an -approximate second-order stationary point in Õ(log n/ 1.75 ) iterations.…”

“…We further assume that the stochastic gradients are Lipschitz (or equivalently, the underlying functions are gradient-Lipschitz, see Assumption 2), which is also adopted in most of the existing works; see e.g. [8,19,20,34]. We demonstrate that a simple extended version of our algorithm takes O(log 2 n) iterations to detect a negative curvature direction using only stochastic gradients, and then obtain an Ω(1) function value decrease with high probability.…”

“…Technically, our work is inspired by the perturbed gradient descent (PGD) algorithm in [19,20] and the perturbed accelerated gradient descent (PAGD) algorithm in [21]. Specifically, PGD applies gradient descents iteratively until it reaches a point with small gradient, which can be a potential saddle point.…”

“…See Proposition 3 and Proposition 5. After escaping the saddle point, similar to PGD and PAGD, we switch back to GD and AGD iterations, which are efficient to decrease the function value when the gradient is large [19,20,21].…”

Nonconvex Stochastic Scaled-Gradient Descent and Generalized Eigenvector Problems

“…Motivated by recent work on escaping saddle points (Ge et al, 2015;Lee et al, 2016;Jin et al, 2019), one can show that SSGD algorithm equipped with the aforementioned artificial noise injection escapes from all saddle points, and hence the initialization condition ( 14) can be dropped. First, we generalize Assumption 2.1 for local convergence to the following for global convergence:…”