A High Probability Analysis of Adaptive SGD with Momentum

Li, Xiaoyu; Orabona, Francesco

doi:10.48550/arxiv.2007.14294

Cited by 7 publications

(11 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…where in the last inequality we used (21). From (22) and (20), with probability at least 1 − δ 1 − δ 2 , we obtain…”

Section: Proof Of Theoremmentioning

confidence: 99%

“…[32] and [33] and [21] gave the high probability convergence guarantees under the Assumption 4. Intuitively, it results in that the tails of the noise distribution are dominated by tails of a Gaussian distribution.…”

Section: Convergence Analysis With High Probabilitymentioning

confidence: 99%

“…[20] demonstrated high probability bounds for PEGASOS algorithm by virtue of Freeman's inequality. A recent study by [21] indicated a high probability analysis for Delayed ADAGRAD with momentum applied to smooth nonconvex objective functions. Different from recently developed theoretical analysis of [13], which has been focused on bounds in expectation for online meta-learning framework, we also propound high probability convergence rates to describe the performance of the proposed algorithm on single runs.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Dynamic Regret Analysis for Online Meta-Learning

Parvin¹,

Khorram²

2021

Preprint

View full text Add to dashboard Cite

The online meta-learning framework has arisen as a powerful tool for the continual lifelong learning setting. The goal for an agent is to quickly learn new tasks by drawing on prior experience, while it faces with tasks one after another. This formulation involves two levels: outer level which learns meta-learners and inner level which learns task-specific models, with only a small amount of data from the current task. While existing methods provide static regret analysis for the online meta-learning framework, we establish performance in terms of dynamic regret which handles changing environments from a global prospective. We also build off of a generalized version of the adaptive gradient methods that covers both ADAM and ADAGRAD to learn meta-learners in the outer level. We carry out our analyses in a stochastic setting, and in expectation prove a logarithmic local dynamic regret which depends explicitly on the total number of iterations T and parameters of the learner. Apart from, we also indicate high probability bounds on the convergence rates of proposed algorithm with appropriate selection of parameters, which have not been argued before.

show abstract

“…where in the last inequality we used (21). From (22) and (20), with probability at least 1 − δ 1 − δ 2 , we obtain…”

Section: Proof Of Theoremmentioning

confidence: 99%

Section: Convergence Analysis With High Probabilitymentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Dynamic Regret Analysis for Online Meta-Learning

Parvin¹,

Khorram²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In particular, almost sure convergence to a first-order stationary point is proved assuming only strong smoothness and a weak assumption on the noise in [35]; mean convergence under the PL inequality is shown in, e.g, [37]. High-probability convergence results assuming strong smoothness and norm sub-Gaussian noise were provided in e.g., [38], and in [39] for strongly convex functions in the non-smooth setting.…”

Section: Introductionmentioning

confidence: 99%

“…We also acknowledge representative prior works on inexact gradient and proximal-gradient methods for batch optimization in, e.g., [32]- [34], and for the stochastic gradient descent in [35]- [38] (see also references therein). In particular, almost sure convergence to a first-order stationary point is proved assuming only strong smoothness and a weak assumption on the noise in [35]; mean convergence under the PL inequality is shown in, e.g, [37].…”

Section: Introductionmentioning

confidence: 99%

Online Stochastic Gradient Methods Under Sub-Weibull Noise and the Polyak-Łojasiewicz Condition

Kim¹,

Madden²,

Dall’Anese³

2021

Preprint

View full text Add to dashboard Cite

This paper focuses on the online gradient and proximal-gradient methods for optimization and learning problems with data streams. The performance of the online gradient descent method is first examined in a setting where the cost satisfies the Polyak-Łojasiewicz (PL) inequality and inexact gradient information is available. Convergence results show that the instantaneous regret converges linearly up to an error that depends on the variability of the problem and the statistics of the gradient error; in particular, we provide bounds in expectation and in high probability (that hold iteration-wise), with the latter derived by leveraging a sub-Weibull model for the errors affecting the gradient. Similar convergence results are then provided for the online proximal-gradient method, under the assumption that the composite cost satisfies the proximal-PL condition. The convergence results are applicable to a number of data processing, learning, and feedback-optimization tasks, where the cost functions may not be strongly convex, but satisfies the PL inequity. In the case of static costs, the bounds provide a new characterization of the convergence of gradient and proximal-gradient methods with a sub-Weibull gradient error. Illustrative simulations are provided for a real-time demand response problem in the context of power systems. S. Kim and L. Madden are with the Department of Applied Mathematics at the University of Colorado Boulder; E. Dall'Anese is with the Department of Electrical, Computer and Energy Engineering and an affiliate faculty with the

show abstract

High Probability Guarantees For Federated Learning

Ramishetty,

Hashemi

2023

2023 59th Annual Allerton Conference on Communication, Control, and Computing (Allerton)

View full text Add to dashboard Cite

A High Probability Analysis of Adaptive SGD with Momentum

Cited by 7 publications

References 6 publications

Dynamic Regret Analysis for Online Meta-Learning

Dynamic Regret Analysis for Online Meta-Learning

Online Stochastic Gradient Methods Under Sub-Weibull Noise and the Polyak-Łojasiewicz Condition

High Probability Guarantees For Federated Learning

Contact Info

Product

Resources

About