Stochastic Intermediate Gradient Method for Convex Problems with Stochastic Inexact Oracle

Dvurechensky, Pavel; Gasnikov, Alexander

doi:10.1007/s10957-016-0999-6

Cited by 65 publications

(56 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The expression in (16) is obtained by adding the upper bounds on k 1 and k 2 in (27) and (29). Now assume that property (ii) holds.…”

Section: Technical Resultsmentioning

confidence: 99%

“…Further to this work, the same authors describe an intermediate gradient method that uses an inexact oracle [26]. That work is extended in [27] to handle the case of composite functions, where a stochastic inexact oracle is also introduced. A method based on inexact dual gradient information is introduced in [28], while [2] considers the minimization of an unconstrained, convex, and composite function, where error is present in the gradient of the smooth term or in the proximity operator for the non-smooth term.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Inexact Coordinate Descent: Complexity and Preconditioning

Tappenden

Richtárik

Gondzio

2016

J Optim Theory Appl

View full text Add to dashboard Cite

One of the key steps at each iteration of a randomized block coordinate descent method consists in determining the update to a block of variables. Existing algorithms assume that in order to compute the update, a particular subproblem is solved exactly. In this work, we relax this requirement and allow for the subproblem to be solved inexactly, leading to an inexact block coordinate descent method. Our approach incorporates the best known results for exact updates as a special case. Moreover, these theoretical guarantees are complemented by practical considerations: the use of iterative techniques to determine the update and the use of preconditioning for further acceleration.

show abstract

“…The expression in (16) is obtained by adding the upper bounds on k 1 and k 2 in (27) and (29). Now assume that property (ii) holds.…”

Section: Technical Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Inexact Coordinate Descent: Complexity and Preconditioning

Tappenden

Richtárik

Gondzio

2016

J Optim Theory Appl

View full text Add to dashboard Cite

show abstract

“…then (21a) represents Nesterov's method (2c). For gradient descent (2a), we can alternatively use ψ t = z t = y t := x t with the corresponding matrices In what follows, we demonstrate how property (19) of the nonlinear mapping ∆ allows us to obtain upper bounds on J when system (21a) is driven by the white stochastic input w t with zero mean and identity covariance. for some matrix Π, let X be a positive semidefinite matrix, and let λ be a nonnegative scalar such that system (21a)…”

Section: General Strongly Convex Problemsmentioning

confidence: 94%

“…Summing up inequalities (37) Similar to the first part of the proof of Lemma 1, we can use LMI (24) and inequality (19) to write…”

Section: Proof Of the Bounds Inmentioning

confidence: 99%

Variance Amplification of Accelerated First-Order Algorithms for Strongly Convex Quadratic Optimization Problems

Mohammadi

Razaviyayn

Jovanović

2018

2018 IEEE Conference on Decision and Control (CDC)

View full text Add to dashboard Cite

We study the robustness of accelerated first-order algorithms to stochastic uncertainties in gradient evaluation.Specifically, for unconstrained, smooth, strongly convex optimization problems, we examine the mean-square error in the optimization variable when the iterates are perturbed by additive white noise. This type of uncertainty may arise in situations where an approximation of the gradient is sought through measurements of a real system or in a distributed computation over network. Even though the underlying dynamics of first-order algorithms for this class of problems are nonlinear, we establish upper bounds on the mean-square deviation from the optimal value that are tight up to constant factors. Our analysis quantifies fundamental trade-offs between noise amplification and convergence rates obtained via any acceleration scheme similar to Nesterov's or heavy-ball methods. To gain additional analytical insight, for strongly convex quadratic problems we explicitly evaluate the steady-state variance of the optimization variable in terms of the eigenvalues of the Hessian of the objective function. We demonstrate that the entire spectrum of the Hessian, rather than just the extreme eigenvalues, influence robustness of noisy algorithms. We specialize this result to the problem of distributed averaging over undirected networks and examine the role of network size and topology on the robustness of noisy accelerated algorithms. the steady-state mean error in the objective value and it was shown that, for the same convergence rate, Nesterov-like method with properly-selected parameters can be more robust than gradient descent. This is not surprising because gradient descent can be viewed as a special case of Nesterov's method with a zero momentum parameter. This observation was used in [41] to design an optimal multi-stage algorithm that does not require information about variance of the noise. In this paper, however, we focus on the variance amplification of the iterates (and not the function value) and discuss the connections between the two robustness measures. We show that any choice of parameters for Nesterov's or heavy-ball methods that yields an accelerated convergence rate increases variance amplification relative to gradient descent. More precisely, for the problem with the condition number κ, an algorithm with accelerated convergence rate of at least 1 − c/ √ κ, where c is a positive constant, increases the variance amplification in the iterates by a factor of √ κ. The robustness problem was also studied in [42] where the authors show similar behavior of Nesterov's method and gradient descent in an asymptotic regime in which the stepsize May 28, 2019 DRAFT 4) We extend our analysis from quadratic objective functions to general strongly convex problems. We borrow an approach based on linear matrix inequalities from control theory to establish upper bounds on the noise amplification of both gradient descent and Nesterov's accelerated algorithm. Furthermore, for any given condition number, we demonstrate th...

show abstract

“…which is extended in [6] and [8], analyzes the convergence of common gradient based-methods when gradient or gradient-type mappings are computed inexactly.…”

Section: Introductionmentioning

confidence: 99%

The Inexact Cyclic Block Proximal Gradient Method and Properties of Inexact Proximal Maps

Maia¹,

Gutman²,

Hughes³

2022

Preprint

View full text Add to dashboard Cite

This paper expands the Cyclic Block Proximal Gradient method for block separable composite minimization by allowing for inexactly computed gradients and proximal maps. The resultant algorithm, the Inexact Cyclic Block Proximal Gradient (I-CBPG) method, shares the same convergence rate as its exactly computed analogue provided the allowable errors decrease sufficiently quickly or are pre-selected to be sufficiently small. We provide numerical experiments that showcase the practical computational advantage of I-CBPG for certain fixed tolerances of approximation error and for a dynamically decreasing error tolerance regime in particular. We establish a tight relationship between inexact proximal map evaluations and δ-subgradients in our δ-Second Prox Theorem. This theorem forms the foundation of our convergence analysis and enables us to show that inexact gradient computations and other notions of inexact proximal map computation can be subsumed within a single unifying framework.

show abstract

Stochastic Intermediate Gradient Method for Convex Problems with Stochastic Inexact Oracle

Cited by 65 publications

References 9 publications

Inexact Coordinate Descent: Complexity and Preconditioning

Inexact Coordinate Descent: Complexity and Preconditioning

Variance Amplification of Accelerated First-Order Algorithms for Strongly Convex Quadratic Optimization Problems

The Inexact Cyclic Block Proximal Gradient Method and Properties of Inexact Proximal Maps

Contact Info

Product

Resources

About