Robust Accelerated Gradient Methods for Smooth Strongly Convex Functions

Aybat, Necdet Serhat; Fallah, Alireza; Gürbüzbalaban, Mert; Ozdaglar, Asuman

doi:10.1137/19m1244925

Cited by 36 publications

(41 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We formalize this problem in Section 4 and propose an adapted H 2 -performance measure. It has already been recognized in different settings [26,2] that H 2 -performance is a measure for the robustness properties of optimization algorithms.…”

Section: Problem Formulationmentioning

confidence: 99%

“…Only recently, several works relying on this systems theoretic view on optimization algorithms have been published [3,17,9,38,34,23,24,10], partly also providing different approaches for convergence rate analysis using techniques from robust control. However, convergence rates are only one side of the coin; in several applications, e.g., in a data-based setting, also robustness with respect to various kinds of disturbances is a key issue and similar analysis tools have been developed [2,26]. In addition, the problem of designing algorithms specifically tailored to classes of structured optimization problems has only been touched upon so far [8,11,18].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Robust and structure exploiting optimisation algorithms: an integral quadratic constraint approach

Michalowsky

Scherer

Ebenbauer

2020

International Journal of Control

View full text Add to dashboard Cite

We consider the problem of analyzing and designing gradient-based discrete-time optimization algorithms for a class of unconstrained optimization problems having strongly convex objective functions with Lipschitz continuous gradient. By formulating the problem as a robustness analysis problem and making use of a suitable adaptation of the theory of integral quadratic constraints, we establish a framework that allows to analyze convergence rates and robustness properties of existing algorithms and enables the design of novel robust optimization algorithms with prespecified guarantees capable of exploiting additional structure in the objective function.

show abstract

Section: Problem Formulationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Robust and structure exploiting optimisation algorithms: an integral quadratic constraint approach

Michalowsky

Scherer

Ebenbauer

2020

International Journal of Control

View full text Add to dashboard Cite

show abstract

“…Remark 1: The performance measure J in (6d) quantifies the steady-state variance of the iterates of first-order algorithms. Robustness of noisy algorithms can be also evaluated using alternative performance measures, e.g., the mean value of the error in the objective function [40],…”

Section: A Influence Of the Eigenvalues Of The Hessian Matrixmentioning

confidence: 99%

“…This work builds on our recent conference papers [38], [39]. In a concurrent work [40], similar approach was used to analyze the robustness of gradient descent and Nesterov's accelerated method. Therein, robustness was defined as goes to zero.…”

Section: Introductionmentioning

confidence: 99%

Variance Amplification of Accelerated First-Order Algorithms for Strongly Convex Quadratic Optimization Problems

Mohammadi

Razaviyayn

Jovanović

2018

2018 IEEE Conference on Decision and Control (CDC)

View full text Add to dashboard Cite

We study the robustness of accelerated first-order algorithms to stochastic uncertainties in gradient evaluation.Specifically, for unconstrained, smooth, strongly convex optimization problems, we examine the mean-square error in the optimization variable when the iterates are perturbed by additive white noise. This type of uncertainty may arise in situations where an approximation of the gradient is sought through measurements of a real system or in a distributed computation over network. Even though the underlying dynamics of first-order algorithms for this class of problems are nonlinear, we establish upper bounds on the mean-square deviation from the optimal value that are tight up to constant factors. Our analysis quantifies fundamental trade-offs between noise amplification and convergence rates obtained via any acceleration scheme similar to Nesterov's or heavy-ball methods. To gain additional analytical insight, for strongly convex quadratic problems we explicitly evaluate the steady-state variance of the optimization variable in terms of the eigenvalues of the Hessian of the objective function. We demonstrate that the entire spectrum of the Hessian, rather than just the extreme eigenvalues, influence robustness of noisy algorithms. We specialize this result to the problem of distributed averaging over undirected networks and examine the role of network size and topology on the robustness of noisy accelerated algorithms. the steady-state mean error in the objective value and it was shown that, for the same convergence rate, Nesterov-like method with properly-selected parameters can be more robust than gradient descent. This is not surprising because gradient descent can be viewed as a special case of Nesterov's method with a zero momentum parameter. This observation was used in [41] to design an optimal multi-stage algorithm that does not require information about variance of the noise. In this paper, however, we focus on the variance amplification of the iterates (and not the function value) and discuss the connections between the two robustness measures. We show that any choice of parameters for Nesterov's or heavy-ball methods that yields an accelerated convergence rate increases variance amplification relative to gradient descent. More precisely, for the problem with the condition number κ, an algorithm with accelerated convergence rate of at least 1 − c/ √ κ, where c is a positive constant, increases the variance amplification in the iterates by a factor of √ κ. The robustness problem was also studied in [42] where the authors show similar behavior of Nesterov's method and gradient descent in an asymptotic regime in which the stepsize May 28, 2019 DRAFT 4) We extend our analysis from quadratic objective functions to general strongly convex problems. We borrow an approach based on linear matrix inequalities from control theory to establish upper bounds on the noise amplification of both gradient descent and Nesterov's accelerated algorithm. Furthermore, for any given condition number, we demonstrate th...

show abstract

“…This allows SAPD to enjoy fast convergence with a robust performance in the presence of stochastic gradient noise. Achieving systematic trade-offs between the rate and robustness have been previously studied in [2] in the context of accelerated methods for smooth strongly convex minimization problems. To our knowledge, our work is the first one that can trade-off ρ with J in a systematic fashion in the context of primal-dual algorithms for solving the SP problem in (1.1).…”

mentioning

confidence: 99%

Robust Accelerated Primal-Dual Methods for Computing Saddle Points

Zhang¹,

Aybat²,

Gürbüzbalaban³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

We consider strongly convex/strongly concave saddle point problems assuming we have access to unbiased stochastic estimates of the gradients. We propose a stochastic accelerated primal-dual (SAPD) algorithm and show that SAPD iterate sequence, generated using constant primal-dual step sizes, linearly converges to a neighborhood of the unique saddle point, where the size of the neighborhood is determined by the asymptotic variance of the iterates. Interpreting the asymptotic variance as a measure of robustness to gradient noise, we obtain explicit characterizations of robustness in terms of SAPD parameters and problem constants. Based on these characterizations, we develop computationally tractable techniques for optimizing the SAPD parameters, i.e., the primal and dual step sizes, and the momentum parameter, to achieve a desired trade-off between the convergence rate and robustness on the Pareto curve. This allows SAPD to enjoy fast convergence properties while being robust to noise as an accelerated method. We also show that SAPD admits convergence guarantees for the gap metric with a variance term optimal up to a logarithmic factor -which can be removed by employing a restarting strategy. Furthermore, to our knowledge, our work is the first one showing an iteration complexity result for the gap function on smooth SCSC problems without the bounded domain assumption. Finally, we illustrate the efficiency of our approach on distributionally robust logistic regression problems.

show abstract

Robust Accelerated Gradient Methods for Smooth Strongly Convex Functions

Cited by 36 publications

References 15 publications

Robust and structure exploiting optimisation algorithms: an integral quadratic constraint approach

Robust and structure exploiting optimisation algorithms: an integral quadratic constraint approach

Variance Amplification of Accelerated First-Order Algorithms for Strongly Convex Quadratic Optimization Problems

Robust Accelerated Primal-Dual Methods for Computing Saddle Points

Contact Info

Product

Resources

About