Variance Amplification of Accelerated First-Order Algorithms for Strongly Convex Quadratic Optimization Problems

Abstract

We study the robustness of accelerated first-order algorithms to stochastic uncertainties in gradient evaluation.Specifically, for unconstrained, smooth, strongly convex optimization problems, we examine the mean-square error in the optimization variable when the iterates are perturbed by additive white noise. This type of uncertainty may arise in situations where an approximation of the gradient is sought through measurements of a real system or in a distributed computation over network. Even though the underlying dynamics of first-order algorithms for this class of problems are nonlinear, we establish upper bounds on the mean-square deviation from the optimal value that are tight up to constant factors. Our analysis quantifies fundamental trade-offs between noise amplification and convergence rates obtained via any acceleration scheme similar to Nesterov's or heavy-ball methods. To gain additional analytical insight, for strongly convex quadratic problems we explicitly evaluate the steady-state variance of the optimization variable in terms of the eigenvalues of the Hessian of the objective function. We demonstrate that the entire spectrum of the Hessian, rather than just the extreme eigenvalues, influence robustness of noisy algorithms. We specialize this result to the problem of distributed averaging over undirected networks and examine the role of network size and topology on the robustness of noisy accelerated algorithms. the steady-state mean error in the objective value and it was shown that, for the same convergence rate, Nesterov-like method with properly-selected parameters can be more robust than gradient descent. This is not surprising because gradient descent can be viewed as a special case of Nesterov's method with a zero momentum parameter. This observation was used in [41] to design an optimal multi-stage algorithm that does not require information about variance of the noise. In this paper, however, we focus on the variance amplification of the iterates (and not the function value) and discuss the connections between the two robustness measures. We show that any choice of parameters for Nesterov's or heavy-ball methods that yields an accelerated convergence rate increases variance amplification relative to gradient descent. More precisely, for the problem with the condition number κ, an algorithm with accelerated convergence rate of at least 1 − c/ √ κ, where c is a positive constant, increases the variance amplification in the iterates by a factor of √ κ. The robustness problem was also studied in [42] where the authors show similar behavior of Nesterov's method and gradient descent in an asymptotic regime in which the stepsize May 28, 2019 DRAFT 4) We extend our analysis from quadratic objective functions to general strongly convex problems. We borrow an approach based on linear matrix inequalities from control theory to establish upper bounds on the noise amplification of both gradient descent and Nesterov's accelerated algorithm. Furthermore, for any given condition number, we demonstrate th...

Robust and structure exploiting optimisation algorithms: an integral quadratic constraint approach

“…The fastest method in terms of convergence rates, the Triple Momentum Method, however, has the worst noise attenuation. We note that these results have to be taken with care since they only provide upper bounds; still, they are qualitatively in accordance with the results from [26], where a similar performance channel has been analyzed for quadratic optimization problems.…”

“…Theorem 4. Let L ≥ m > 0 and let ∆ ρ (m, L) be defined as in (26). Then, for each ρ ∈ (0, 1], ∆ ρ satisfies the IQC defined by Π for each ∆ ρ ∈ ∆ ρ (m, L) and each…”

Robustness of Accelerated First-Order Algorithms for Strongly Convex Optimization Problems

“…These inequalities in conjunction with ( 16) yield ( ψ t / ψ 0 ) 2 ≤ λ max (P )/λ min (P ). Finally, (15) follows from combining this inequality with A t = (A T ) t = sup ψ 0 = 0 ψ t / ψ 0 .…”

Inducing Uniform Asymptotic Stability in Non-Autonomous Accelerated Optimization Dynamics via Hybrid Regularization

“…Moreover, it was also recently shown in [11] that for the case when (p) = 2p+1 and h(x) = 0.5|x| 2 , Runge-Kutta discretization methods applied to (2) can generate discrete-time algorithms that achieve acceleration. While these results have been instrumental in the analysis and design of various optimization algorithms with provable acceleration and convergence properties, the study of the robustness properties of these algorithms has been considered only recently [12], [13], [14], [5], [15]. Indeed, as it has been noted in the literature, e.g., [16], [8], dynamics of the form (2) may become unstable under small disturbances or even under forward Euler discretization.…”