Simple and optimal high-probability bounds for strongly-convex stochastic gradient descent

Harvey, Nicholas J. A.; Liaw, Christopher; Randhawa, Sikander

doi:10.48550/arxiv.1909.00843

Cited by 6 publications

(7 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Convergence results are derived for an empirical risk that is smooth and satisfies the PL inequality, and under a bounded variation property (related to the so-called volatility [5]). As explained in Section 1.2, prior works provided high-probability convergence bounds under the assumption of Lipschitz continuity and strong convexity of the loss function [6,7,8,9]. In this paper, we prove convergence bounds matching the optimal bounds of [8], but without strong convexity.…”

Section: Introductionmentioning

confidence: 76%

“…Regarding convergence of SGD, other papers prove high probability convergence bounds for projected gradient descent assuming Lipschitz continuity and strong convexity [6,7,8,9]. A function cannot be globally Lipschitz continuous and strongly convex (unless it is essentially a constant function), which is why they consider projected gradient.…”

Section: Prior Work and Contributionsmentioning

confidence: 99%

See 1 more Smart Citation

High-probability Convergence Bounds for Non-convex Stochastic Gradient Descent

Madden¹,

Dall’Anese²,

Becker³

2020

Preprint

View full text Add to dashboard Cite

Stochastic gradient descent (with a mini-batch) is one of the most common iterative algorithms used in machine learning. While being computationally cheap to implement, recent literature suggests that it may also have implicit regularization properties that prevent overfitting. This paper analyzes the properties of stochastic gradient descent from a theoretical standpoint to help bridge the gap between theoretical and empirical results; in particular, we prove bounds that depend explicitly on the number of epochs. Assuming smoothness, the Polyak-Łojasiewicz inequality, and the bounded variation property, we prove high probability bounds on the convergence rate. Assuming Lipschitz continuity and smoothness, we prove high probability bounds on the uniform stability. Putting these together (noting that some of the assumptions imply each other), we bound the true risk of the iterates of stochastic gradient descent. For convergence, our high probability bounds match existing expected bounds. For stability, our high probability bounds extend the nonconvex expected bound in Hardt et al. (2015). We use existing results to bound the generalization error using the stability. Finally, we put the convergence and generalization bounds together. We find that for a certain number of epochs of stochastic gradient descent, the convergence and generalization balance and we get a true risk bound that goes to zero as the number of samples goes to infinity.Preprint. Under review.

show abstract

Section: Introductionmentioning

confidence: 76%

Section: Prior Work and Contributionsmentioning

confidence: 99%

High-probability Convergence Bounds for Non-convex Stochastic Gradient Descent

Madden¹,

Dall’Anese²,

Becker³

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…The lemma was proposed by Klein and Young [2015] (see lemma 4) to show the tightness of the Chernoff bound. Recently it has been used to derive a high probability lower bound of the 1/t step-size [Harvey et al, 2019b]. We will now use Lemma 7.2 to prove the following high probability lower bound of the step decay scheme in Algorithm 2 for S = T / log α T .…”

Section: Proofs Of Sectionmentioning

confidence: 99%

On the Convergence of Step Decay Step-Size for Stochastic Optimization

Wang,

Magnússon,

Johansson

2021

Preprint

View full text Add to dashboard Cite

The convergence of stochastic gradient descent is highly dependent on the step-size, especially on non-convex problems such as neural network training. Step decay step-size schedules (constant and then cut) are widely used in practice because of their excellent convergence and generalization qualities, but their theoretical properties are not yet well understood. We provide the convergence results for step decay in the non-convex regime, ensuring that the gradient norm vanishes at an O(ln T / √ T ) rate. We also provide the convergence guarantees for general (possibly non-smooth) convex problems, ensuring an O(ln T / √ T ) convergence rate. Finally, in the strongly convex case, we establish an O(ln T /T ) rate for smooth problems, which we also prove to be tight, and an O(ln 2 T /T ) rate without the smoothness assumption. We illustrate the practical efficiency of the step decay step-size in several large scale deep neural network training tasks.

show abstract

“…Kakade & Tewari (2009) used Freeman's inequality to prove high probability bounds for an algorithm solving the SVM objective function. For classic SGD, Harvey et al (2019b) and Harvey et al (2019a) used a generalized Freedmans inequality to prove bounds in non-smooth and strongly convex case, while Jain et al (2019) proved the optimal bound for the last iterate of SGD with high probability. As far as we know, there are currently no high probability bounds for adaptive methods in the nonconvex setting.…”

Section: Related Workmentioning

confidence: 99%

A High Probability Analysis of Adaptive SGD with Momentum

Li,

Orabona

2020

Preprint

View full text Add to dashboard Cite

Stochastic Gradient Descent (SGD) and its variants are the most used algorithms in machine learning applications. In particular, SGD with adaptive learning rates and momentum is the industry standard to train deep networks. Despite the enormous success of these methods, our theoretical understanding of these variants in the nonconvex setting is not complete, with most of the results only proving convergence in expectation and with strong assumptions on the stochastic gradients. In this paper, we present a high probability analysis for adaptive and momentum algorithms, under weak assumptions on the function, stochastic gradients, and learning rates. We use it to prove for the first time the convergence of the gradients to zero in high probability in the smooth nonconvex setting for Delayed AdaGrad with momentum.

show abstract

Simple and optimal high-probability bounds for strongly-convex stochastic gradient descent

Cited by 6 publications

References 0 publications

High-probability Convergence Bounds for Non-convex Stochastic Gradient Descent

High-probability Convergence Bounds for Non-convex Stochastic Gradient Descent

On the Convergence of Step Decay Step-Size for Stochastic Optimization

A High Probability Analysis of Adaptive SGD with Momentum

Contact Info

Product

Resources

About