Lower Bounds for Finding Stationary Points I

Carmon, Yair; Duchi, John C.; Hinder, Oliver; Sidford, Aaron

doi:10.48550/arxiv.1710.11606

Cited by 28 publications

(87 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Optimal rates for finding second-order stationary points. Carmon et al [2017a] have presented lower bounds that imply that GD achieves the optimal rate for finding stationary points for gradient Lipschitz functions. In our setting, we additionally assume that the Hessian is Lipschitz.…”

Section: Discussionmentioning

confidence: 99%

On Nonconvex Optimization for Machine Learning: Gradients, Stochasticity, and Saddle Points

Jin,

Netrapalli,

et al. 2019

Preprint

View full text Add to dashboard Cite

Gradient descent (GD) and stochastic gradient descent (SGD) are the workhorses of large-scale machine learning. While classical theory focused on analyzing the performance of these methods in convex optimization problems, the most notable successes in machine learning have involved nonconvex optimization, and a gap has arisen between theory and practice. Indeed, traditional analyses of GD and SGD show that both algorithms converge to stationary points efficiently. But these analyses do not take into account the possibility of converging to saddle points. More recent theory has shown that GD and SGD can avoid saddle points, but the dependence on dimension in these analyses is polynomial. For modern machine learning, where the dimension can be in the millions, such dependence would be catastrophic. We analyze perturbed versions of GD and SGD and show that they are truly efficient-their dimension dependence is only polylogarithmic. Indeed, these algorithms converge to second-order stationary points in essentially the same time as they take to converge to classical first-order stationary points.

show abstract

Section: Discussionmentioning

confidence: 99%

On Nonconvex Optimization for Machine Learning: Gradients, Stochasticity, and Saddle Points

Jin,

Netrapalli,

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…The iteration complexity order O(ε −2 log(ε −1 )) in Theorem 3.4 is tight (up to logarithmic factors). This is due to the fact that for general non-convex smooth problems, finding an ε-stationary solution requires at least Ω(ε −2 ) gradient evaluations [7,47]. Clearly, this lower bound is also valid for finding an ε-FNE of PL-games.…”

Section: Convergence Analysis Of Multi-step Gradient Descent Ascent A...mentioning

confidence: 99%

Solving a Class of Non-Convex Min-Max Games Using Iterative First Order Methods

Nouiehed,

Sanjabi,

Huang

et al. 2019

Preprint

View full text Add to dashboard Cite

Recent applications that arise in machine learning have surged significant interest in solving min-max saddle point games. This problem has been extensively studied in the convex-concave regime for which a global equilibrium solution can be computed efficiently. In this paper, we study the problem in the non-convex regime and show that an ε-first order stationary point of the game can be computed when one of the player's objective can be optimized to global optimality efficiently. In particular, we first consider the case where the objective of one of the players satisfies the Polyak-Łojasiewicz (PL) condition. For such a game, we show that a simple multi-step gradient descent-ascent algorithm finds an ε-first order stationary point of the problem in O(ε −2 ) iterations. Then we show that our framework can also be applied to the case where the objective of the "max-player" is concave. In this case, we propose a multi-step gradient descent-ascent algorithm that finds an ε-first order stationary point of the game in O(ε −3.5 ) iterations, which is the best known rate in the literature. We applied our algorithm to a fair classification problem of Fashion-MNIST dataset and observed that the proposed algorithm results in smoother training and better generalization.

show abstract

“…We consider intermittent communication algorithms, which attempt to optimize F using M parallel workers, each of which is allowed K queries to g in each of R rounds of communication. Such intermittent communication algorithms can be formalized using the graph oracle framework of Woodworth et al (2018) which focuses on the dependence structure between different stochastic gradient computations, and to facilitate our lower bounds, we follow Carmon et al (2017) and focus our attention on distributed zero-respecting algorithms:…”

Section: Setting and Notationmentioning

confidence: 99%

“…Furthermore, work in other contexts has succeeded in proving that the min-max complexity for arbitrary randomized algorithms often matches the min-max complexity for zero-respecting algorithms, but at the expense of much more complicated proofs (e.g. Woodworth and Srebro, 2016;Carmon et al, 2017;Arjevani et al, 2020).…”

Section: Setting and Notationmentioning

confidence: 99%

The Min-Max Complexity of Distributed Stochastic Convex Optimization with Intermittent Communication

Woodworth,

Bullins,

Shamir

et al. 2021

Preprint

View full text Add to dashboard Cite

We resolve the min-max complexity of distributed stochastic convex optimization (up to a log factor) in the intermittent communication setting, where M machines work in parallel over the course of R rounds of communication to optimize the objective, and during each round of communication, each machine may sequentially compute K stochastic gradient estimates. We present a novel lower bound with a matching upper bound that establishes an optimal algorithm.

show abstract

Lower Bounds for Finding Stationary Points I

Cited by 28 publications

References 0 publications

On Nonconvex Optimization for Machine Learning: Gradients, Stochasticity, and Saddle Points

On Nonconvex Optimization for Machine Learning: Gradients, Stochasticity, and Saddle Points

Solving a Class of Non-Convex Min-Max Games Using Iterative First Order Methods

The Min-Max Complexity of Distributed Stochastic Convex Optimization with Intermittent Communication

Contact Info

Product

Resources

About