Finding Approximate Local Minima Faster than Gradient Descent

Agarwal, Naman; Allen-Zhu, Zeyuan; Bullins, Brian; Hazan, Elad; Ma, Tengyu

doi:10.48550/arxiv.1611.01146

Cited by 7 publications

(23 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Reference Oracle Iterations Simplicity Non-stochastic [1,6] Hessian-vector product Õ(log n/ It is worth highlighting that our gradient-descent based algorithm enjoys the following nice features:…”

Section: Settingmentioning

confidence: 99%

“…Similarly to Algorithm 1 and Algorithm 2, the renormalization step Line 6 in Algorithm 3 only guarantees that the value y t would not scales exponentially during the algorithm, and does not affect z 0 ← 0; for t = 1, ..., T s do Sample θ (1) , θ (2) , • • • , θ (m) ∼ D;…”

Section: C2 Proof Details Of Escaping Saddle Points Using Algorithmmentioning

confidence: 99%

“…On the contrary, algorithms with nested loops often suffer from significant overheads in large scales, or introduce concerns with the setting of hyperparameters and numerical stability (see e.g. [1,6]), making them relatively hard to find practical implementations.…”

Section: Introductionmentioning

confidence: 99%

“…[3,24,29] demonstrate that an -approximate second-order stationary point can be find using gradients in Õ(log n/ 1.75 ) iterations. Their results are based on previous works [1,6] using Hessian-vector products and the observation that the Hessian-vector product can be approximated via the difference of two gradient queries. Hence, their implementations contain nested-loop structures with relatively large numbers of hyperparameters.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Escape saddle points by a simple gradient-descent based algorithm

Zhang¹,

2021

Preprint

View full text Add to dashboard Cite

Escaping saddle points is a central research topic in nonconvex optimization. In this paper, we propose a simple gradient-based algorithm such that for a smooth function f : R n → R, it outputs an -approximate second-order stationary point in Õ(log n/ 1.75 ) iterations. Compared to the previous state-of-the-art algorithms by Jin et al. with Õ(log 4 n/ 2 ) or Õ(log 6 n/ 1.75 ) iterations, our algorithm is polynomially better in terms of log n and matches their complexities in terms of 1/ . For the stochastic setting, our algorithm outputs an -approximate second-order stationary point in Õ(log 2 n/ 4 ) iterations. Technically, our main contribution is an idea of implementing a robust Hessian power method using only gradients, which can find negative curvature near saddle points and achieve the polynomial speedup in log n compared to the perturbed gradient descent methods. Finally, we also perform numerical experiments that support our results.

show abstract

Section: Settingmentioning

confidence: 99%

Section: C2 Proof Details Of Escaping Saddle Points Using Algorithmmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Escape saddle points by a simple gradient-descent based algorithm

Zhang¹,

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Queries Oracle [24,56] O(1/ǫ 1.5 ) Hessian [1,15] Õ(log n/ǫ 1.75 ) Hessian-vector product [42,43] Õ(log 4 n/ǫ 2 ) Gradient [44] Õ(log 6 n/ǫ 1.75 ) Gradient this work Õ(log 2 n/ǫ 1.75 ) Quantum evaluation Table 1: A summary of the state-of-the-art works on finding approximate second-order stationary points using different oracles. The query complexities are highlighted in terms of the dimension n and the error ǫ.…”

Section: Referencementioning

confidence: 99%

Quantum algorithms for escaping from saddle points

Zhang,

Leng,

2020

Preprint

View full text Add to dashboard Cite

We initiate the study of quantum algorithms for escaping from saddle points with provable guarantee. Given a function f : R n → R, our quantum algorithm outputs an ǫ-approximate second-order stationary point using Õ(log 2 n/ǫ 1.75 ) 1 queries to the quantum evaluation oracle (i.e., the zeroth-order oracle). Compared to the classical state-of-the-art algorithm by Jin et al. with Õ(log 6 n/ǫ 1.75 ) queries to the gradient oracle (i.e., the first-order oracle), our quantum algorithm is polynomially better in terms of n and matches its complexity in terms of 1/ǫ. Our quantum algorithm is built upon two techniques: First, we replace the classical perturbations in gradient descent methods by simulating quantum wave equations, which constitutes the polynomial speedup in n for escaping from saddle points. Second, we show how to use a quantum gradient computation algorithm due to Jordan to replace the classical gradient queries by quantum evaluation queries with the same complexity. Finally, we also perform numerical experiments that support our quantum speedup.

show abstract

Newton-type methods for non-convex optimization under inexact Hessian information

2019

View full text Add to dashboard Cite

We consider variants of trust-region and adaptive cubic regularization methods for non-convex optimization, in which the Hessian matrix is approximated. Under certain condition on the inexact Hessian, and using approximate solution of the corresponding sub-problems, we provide iteration complexity to achieve ǫ-approximate second-order optimality which have been shown to be tight. Our Hessian approximation condition offers a range of advantages as compared with the prior works and allows for direct construction of the approximate Hessian with a priori guarantees through various techniques, including randomized sampling methods. In this light, we consider the canonical problem of finite-sum minimization, provide appropriate uniform and non-uniform sub-sampling strategies to construct such Hessian approximations, and obtain optimal iteration complexity for the corresponding subsampled trust-region and adaptive cubic regularization methods.

show abstract

Finding Approximate Local Minima Faster than Gradient Descent

Cited by 7 publications

References 0 publications

Escape saddle points by a simple gradient-descent based algorithm

Escape saddle points by a simple gradient-descent based algorithm

Quantum algorithms for escaping from saddle points

Newton-type methods for non-convex optimization under inexact Hessian information

Contact Info

Product

Resources

About