Contracting Proximal Methods for Smooth Convex Optimization

Doikov, Nikita; Nesterov, Yurii

doi:10.1137/19m130769x

Cited by 19 publications

(18 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This step is optional. Moreover, looking at algorithm (6), one may think that we are forgetting the points xk+1 when the function value is increasing:…”

Section: Conceptual Contracting − Point Method Imentioning

confidence: 99%

“…Motivation In the last years, we can see an increasing interest in new frameworks for derivation and justification of different methods for Convex Optimization, provided with a worst-case complexity analysis (see, for example, [3,4,6,11,14,15,18,[20][21][22]). It appears that the accelerated proximal tensor methods [2,20] can be naturally explained through the framework of high-order proximal-point schemes [21] requiring solution of nontrivial auxiliary problem at every iteration.…”

Section: Mathematics Subject Classification 90c25 • 90c06 • 65k05 1 Introductionmentioning

confidence: 99%

“…The results of this work can be also seen as an affine-invariant counterpart of Contracting Proximal Methods from [6]. In the latter algorithms, one needs to fix the prox function which is suitable for the geometry of the problem, in advance.…”

Section: Mathematics Subject Classification 90c25 • 90c06 • 65k05 1 Introductionmentioning

confidence: 99%

“…At the same time, it is known that in Euclidean case, the prox-type methods can be accelerated, achieving O(1/k p+1 ) global rate of convergence [2,6,20,21]. Using additional one-dimensional search at each iteration, this rate can be improved up to…”

mentioning

confidence: 99%

See 3 more Smart Citations

Affine-invariant contracting-point methods for Convex Optimization

Doikov

Nesterov

2022

Math. Program.

Self Cite

View full text Add to dashboard Cite

In this paper, we develop new affine-invariant algorithms for solving composite convex minimization problems with bounded domain. We present a general framework of Contracting-Point methods, which solve at each iteration an auxiliary subproblem restricting the smooth part of the objective function onto contraction of the initial domain. This framework provides us with a systematic way for developing optimization methods of different order, endowed with the global complexity bounds. We show that using an appropriate affine-invariant smoothness condition, it is possible to implement one iteration of the Contracting-Point method by one step of the pure tensor method of degree $$p \ge 1$$ p ≥ 1 . The resulting global rate of convergence in functional residual is then $${\mathcal {O}}(1 / k^p)$$ O ( 1 / k p ) , where k is the iteration counter. It is important that all constants in our bounds are affine-invariant. For $$p = 1$$ p = 1 , our scheme recovers well-known Frank–Wolfe algorithm, providing it with a new interpretation by a general perspective of tensor methods. Finally, within our framework, we present efficient implementation and total complexity analysis of the inexact second-order scheme $$(p = 2)$$ ( p = 2 ) , called Contracting Newton method. It can be seen as a proper implementation of the trust-region idea. Preliminary numerical results confirm its good practical performance both in the number of iterations, and in computational time.

show abstract

“…This step is optional. Moreover, looking at algorithm (6), one may think that we are forgetting the points xk+1 when the function value is increasing:…”

Section: Conceptual Contracting − Point Method Imentioning

confidence: 99%

Section: Mathematics Subject Classification 90c25 • 90c06 • 65k05 1 Introductionmentioning

confidence: 99%

Section: Mathematics Subject Classification 90c25 • 90c06 • 65k05 1 Introductionmentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

Affine-invariant contracting-point methods for Convex Optimization

Doikov

Nesterov

2022

Math. Program.

Self Cite

View full text Add to dashboard Cite

show abstract

“…In [3], it is proved that the convergence rate of the objective function values is o(k −2 ), and the iterates {x k } converge to a minimizer of this problem as α > 3. Recently, Doikov and Nesterov [18] presented a new accelerated algorithm for solving such problem, in which they used the high-order tensor methods to solve the inner subproblems and gave a complexity estimate under some assumptions. For the case with Lipschitz continuously differentiable but nonconvex loss function, Wen, Chen and Pong [35] proved that the iterates and objective function values generated by the proximal gradient algorithm with extrapolation are R-linearly convergent under the error bound condition.…”

Section: Introductionmentioning

confidence: 99%

Accelerated iterative hard thresholding algorithm for $$l_0$$ regularized regression problem

Bian

2019

J Glob Optim

View full text Add to dashboard Cite

We investigate a class of constrained sparse regression problem with cardinality penalty, where the feasible set is defined by box constraint, and the loss function is convex, but not necessarily smooth. First, we put forward a smoothing fast iterative hard thresholding (SFIHT) algorithm for solving such optimization problems, which combines smoothing approximations, extrapolation techniques and iterative hard thresholding methods. The extrapolation coefficients can be chosen to satisfy sup k β k = 1 in the proposed algorithm. We discuss the convergence behavior of the algorithm with different extrapolation coefficients, and give sufficient conditions to ensure that any accumulation point of the iterates is a local minimizer of the original cardinality penalized problem. In particular, for a class of fixed extrapolation coefficients, we discuss several different update rules of the smoothing parameter and obtain the convergence rate of O(ln k/k) on the loss and objective function values. Second, we consider the case in which the loss function is Lipschitz continuously differentiable, and develop a fast iterative hard thresholding (FIHT) algorithm to solve it. We prove that the iterates of FIHT converge to a local minimizer of the problem that satisfies a desirable lower bound property. Moreover, we show that the convergence rate of loss and objective function values are o(k −2 ). Finally, some numerical examples are presented to illustrate the theoretical results.Keywords cardinality penalty • smoothing method • accelerated algorithm • extrapolation • convergence rate • local minimizer

show abstract

Near-Optimal Hyperfast Second-Order Method for Convex Optimization

Kamzolov

2020

Communications in Computer and Information Science

View full text Add to dashboard Cite

Contracting Proximal Methods for Smooth Convex Optimization

Cited by 19 publications

References 14 publications

Affine-invariant contracting-point methods for Convex Optimization

Affine-invariant contracting-point methods for Convex Optimization

Accelerated iterative hard thresholding algorithm for $$l_0$$ regularized regression problem

Near-Optimal Hyperfast Second-Order Method for Convex Optimization

Contact Info

Product

Resources

About