Online Adaptive Methods, Universality and Acceleration

Levy, Kfir Y.; Yurtsever, Alp; Cevher, Volkan

doi:10.48550/arxiv.1809.02864

Cited by 5 publications

(6 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, have analyzed a choice of adaptive stepsizes similar to the global stepsizes we consider, but their result in the convex setting requires the norm of the gradients strictly greater than zero. Levy et al (2018) propose an acceleration method with adaptive stepsizes which are also similar to our global ones, proving the Õ(1/T 2 ) convergence in the deterministic smooth case and Õ(1/ √ T ) in both general deterministic case and stochastic smooth case, but requiring a bounded-domain assumption.…”

Section: Related Workmentioning

confidence: 85%

On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizes

Li,

Orabona

2018

Preprint

View full text Add to dashboard Cite

Stochastic gradient descent is the method of choice for large scale optimization of machine learning objective functions. Yet, its performance is greatly variable and heavily depends on the choice of the stepsizes. This has motivated a large body of research on adaptive stepsizes. However, there is currently a gap in our theoretical understanding of these methods, especially in the non-convex setting. In this paper, we start closing this gap: we theoretically analyze in the convex and non-convex settings a generalized version of the AdaGrad stepsizes. We show sufficient conditions for these stepsizes to achieve almost sure asymptotic convergence of the gradients to zero, proving the first guarantee for generalized AdaGrad stepsizes in the nonconvex setting. Moreover, we show that these stepsizes allow to automatically adapt to the level of noise of the stochastic gradients in both the convex and non-convex settings, interpolating between O(1/T ) and O(1/ √ T ), up to logarithmic terms.

show abstract

Section: Related Workmentioning

confidence: 85%

On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizes

Li,

Orabona

2018

Preprint

View full text Add to dashboard Cite

show abstract

“…An alternative way which uses the norm of the current (sub)gradient to define the step-size was initiated probably by [219] and became very popular in stochastic optimization for machine learning after the paper [220]. On this avenue it was possible to obtain for ν ∈ {0, 1} universal accelerated optimization method [221] and universal methods for variational inequalities and saddle-point problems [222,223].…”

Section: Connection Between Accelerated Methods and Conditional Gradientmentioning

confidence: 99%

First-Order Methods for Convex Optimization

Dvurechensky¹,

Staudigl²,

Shtern³

2021

Preprint

View full text Add to dashboard Cite

First-order methods for solving convex optimization problems have been at the forefront of mathematical optimization in the last 20 years. The rapid development of this important class of algorithms is motivated by the success stories reported in various applications, including most importantly machine learning, signal processing, imaging and control theory. First-order methods have the potential to provide low accuracy solutions at low computational complexity which makes them an attractive set of tools in large-scale optimization problems. In this survey we cover a number of key developments in gradient-based optimization methods. This includes non-Euclidean extensions of the classical proximal gradient method, and its accelerated versions. Additionally we survey recent developments within the class of projection-free methods, and proximal versions of primal-dual schemes. We give complete proofs for various key results, and highlight the unifying aspects of several optimization algorithms.

show abstract

“…MAG utilizes an MLMC gradient estimation technique that effectively transduces bias into variance; that is, our gradient estimator enjoys a low bias, but exhibits a high variance that depends on the mixing time (see Lemma 3.1). To cope with this dependence, we then resort to AdaGrad (Duchi et al, 2011), which is known to implicitly adapt to the variance of the stochastic gradients (Levy et al, 2018). We describe MAG in Algorithm 1, and state its main convergence guarantee for convex objectives in Theorem 3.3.…”

Section: Mag: Combining Multi-level Monte Carlo Gradient Estimation W...mentioning

confidence: 99%

Adapting to Mixing Time in Stochastic Optimization with Markovian Data

Dorfman¹,

Levy²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

We consider stochastic optimization problems where data is drawn from a Markov chain. Existing methods for this setting crucially rely on knowing the mixing time of the chain, which in real-world applications is usually unknown. We propose the first optimization method that does not require the knowledge of the mixing time, yet obtains the optimal asymptotic convergence rate when applied to convex problems. We further show that our approach can be extended to: (i) finding stationary points in non-convex optimization with Markovian data, and (ii) obtaining better dependence on the mixing time in temporal difference (TD) learning; in both cases, our method is completely oblivious to the mixing time. Our method relies on a novel combination of multi-level Monte Carlo (MLMC) gradient estimation together with an adaptive learning method.

show abstract

Online Adaptive Methods, Universality and Acceleration

Cited by 5 publications

References 19 publications

On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizes

On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizes

First-Order Methods for Convex Optimization

Adapting to Mixing Time in Stochastic Optimization with Markovian Data

Contact Info

Product

Resources

About