On the worst-case complexity of the gradient method with exact line search for smooth strongly convex functions

Klerk, Etienne de; Glineur, François; Taylor, Adrien

doi:10.1007/s11590-016-1087-4

Cited by 55 publications

(89 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…-In [12, Remark 3.1], Drori remarked that the lower bound for smooth convex unconstrained minimization was achieved by a greedy method (referenced to as the ideal first-order method), -In [6, Section 4.1], de Klerk et al study the worst-case complexity of steepest descent with exact line search applied to strongly convex functions. As suggested by an anonymous referee in [6], the worst-case certificates were also valid for the gradient method with an appropriate fixed-step size.…”

Section: Links With Subspace-search Methodsmentioning

confidence: 95%

“…The analysis below is based on the performance estimation methodology which was first introduced in [13] and has been successfully applied to analyze methods in a wide range of settings, including smooth and nonsmooth minimization [14,22,23,50], proximal gradient methods [48,51], saddle-point problems [11] and more recently to operator splitting methods [43]. Here we build upon an approach developed for the analysis of line-searching methods [6], and improve it by providing a tightness proof under some mild conditions. Clearly, a meaningful analysis can only be attained by making some assumptions on the structure of the problem: namely, that f belongs to some given class of functions F and that the initial point x 0 satisfies some conditions.…”

Section: Estimating the Worst-case Performance Of Gfommentioning

confidence: 99%

See 1 more Smart Citation

Efficient first-order methods for convex minimization: a constructive approach

Drori

Taylor

2019

Math. Program.

Self Cite

View full text Add to dashboard Cite

We describe a novel constructive technique for devising efficient first-order methods for a wide range of large-scale convex minimization settings, including smooth, non-smooth, and strongly convex minimization. The technique builds upon a certain variant of the conjugate gradient method to construct a family of methods such that a) all methods in the family share the same worst-case guarantee as the base conjugate gradient method, and b) the family includes a fixed-step first-order method. We demonstrate the effectiveness of the approach by deriving optimal methods for the smooth and non-smooth cases, including new methods that forego knowledge of the problem parameters at the cost of a one-dimensional line search per iteration, and a universal method for the union of these classes that requires a three-dimensional search per iteration. In the strongly convex case, we show how numerical tools can be used to perform the construction, and show that the resulting method offers an improved worst-case bound compared to Nesterov's celebrated fast gradient method. IntroductionConvex optimization plays a central role in many fields of applications, including optimal control, machine learning and signal processing. In particular, when a large number of variables are involved within a convex optimization problem, the use of first-order methods is more and more widespread due to their typically very attractive low computational cost per iteration. This low computational cost comes, however, at a price: first-order methods often suffer from potentially slow convergence speeds, making them appropriate mostly for obtaining low to medium accuracy solutions. Nevertheless, first-order methods remain the methods of choice in many applications and currently receive a lot of attention from the optimization community, which constantly aims at improving them.An effective and fruitful approach used for analyzing and comparing first-order methods is the study of their worst-case behavior through the black-box model. In this setting, methods are only allowed to gain information on the objective through an oracle, which provides the value and the gradient of the objective at selected points.

show abstract

Section: Links With Subspace-search Methodsmentioning

confidence: 95%

Section: Estimating the Worst-case Performance Of Gfommentioning

confidence: 99%

Efficient first-order methods for convex minimization: a constructive approach

Drori

Taylor

2019

Math. Program.

Self Cite

View full text Add to dashboard Cite

show abstract

“…For the sake of clarity and completeness, we start by proving it using the same technique as for the subsequent results (on residual gradient norm and objective function accuracy). The proof methodology relies from the performance estimation methodology (see [9,10,11,14,17]). This technique has the advantage of being transparent and of explicitly identifying weaker assumptions for obtaining this convergence property (see discussions below Theorem 3.1).…”

Section: Upper Bounds On the Global Convergence Ratesmentioning

confidence: 99%

Exact Worst-Case Convergence Rates of the Proximal Gradient Method for Composite Convex Minimization

Taylor

Hendrickx

Glineur

2018

J Optim Theory Appl

Self Cite

View full text Add to dashboard Cite

We study the worst-case convergence rates of the proximal gradient method for minimizing the sum of a smooth strongly convex function and a non-smooth convex function, whose proximal operator is available.We establish the exact worst-case convergence rates of the proximal gradient method in this setting for any step size and for different standard performance measures: objective function accuracy, distance to optimality and residual gradient norm.The proof methodology relies on recent developments in performance estimation of first-order methods, based on semidefinite programming. In the case of the proximal gradient method, this methodology allows obtaining exact and non-asymptotic worst-case guarantees, that are conceptually very simple, although apparently new.On the way, we discuss how strong convexity can be replaced by weaker assumptions, while preserving the corresponding convergence rates. We also establish that the same fixed step size policy is optimal for all three performance measures. Finally, we extend recent results on the worst-case behavior of gradient descent with exact line search to the proximal case.

show abstract

“…There has been a resurging interest in first-order methods using sDR-oracle [17,7,14]. One of the reasons is that, in practice, small-dimensional relaxation allows local adaptation to the curvature of the objective function, which can dramatically improve the practical convergence rate, the classical example being conjugate gradient methods.…”

Section: Introductionmentioning

confidence: 99%

Primal–dual accelerated gradient methods with small-dimensional relaxation oracle

Nesterov

Gasnikov

Guminov

et al. 2020

Optimization Methods and Software

View full text Add to dashboard Cite

In this paper, a new variant of accelerated gradient descent is proposed. The proposed method does not require any information about the objective function, uses exact line search for the practical accelerations of convergence, converges according to the well-known lower bounds for both convex and non-convex objective functions, possesses primal-dual properties and can be applied in the non-euclidian set-up. As far as we know this is the first such method possessing all of the above properties at the same time. We also present a universal version of the method which is applicable to non-smooth problems. We demonstrate how in practice one can efficiently use the combination of line-search and primal-duality by considering a convex optimization problem with a simple structure (for example, linearly constrained).

show abstract

On the worst-case complexity of the gradient method with exact line search for smooth strongly convex functions

Cited by 55 publications

References 5 publications

Efficient first-order methods for convex minimization: a constructive approach

Efficient first-order methods for convex minimization: a constructive approach

Exact Worst-Case Convergence Rates of the Proximal Gradient Method for Composite Convex Minimization

Primal–dual accelerated gradient methods with small-dimensional relaxation oracle

Contact Info

Product

Resources

About