Minimizing Uniformly Convex Functions by Cubic Regularization of Newton Method

Doikov, Nikita; Nesterov, Yurii

doi:10.1007/s10957-021-01838-7

Cited by 16 publications

(14 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…From the uniform convexity of the regularizer (see, e.g. Lemma 2.5 in Doikov and Nesterov (2021)) we can bound the size of the solution,…”

Section: Proofmentioning

confidence: 99%

Lower Complexity Bounds for Minimizing Regularized Functions

Doikov¹

2022

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper, we establish lower bounds for the oracle complexity of the first-order methods minimizing regularized convex functions. We consider the composite representation of the objective. The smooth part has Hölder continuous gradient of degree ν ∈ [0, 1] and is accessible by a black-box local oracle. The composite part is a power of a norm. We prove that the best possible rate for the first-order methods in the large-scale setting for Euclidean norms is of the order O(k −p(1+3ν)/(2(p−1−ν)) ) for the functional residual, where k is the iteration counter and p is the power of regularization. Our formulation covers several cases, including computation of the Cubically regularized Newton step by the first-order gradient methods, in which case the rate becomes O(k −6 ). It can be achieved by the Fast Gradient Method. Thus, our result proves the latter rate to be optimal. We also discover lower complexity bounds for non-Euclidean norms.

show abstract

“…From the uniform convexity of the regularizer (see, e.g. Lemma 2.5 in Doikov and Nesterov (2021)) we can bound the size of the solution,…”

Section: Proofmentioning

confidence: 99%

Lower Complexity Bounds for Minimizing Regularized Functions

Doikov¹

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…This function is uniformly convex of degree p + 1 with parameter σ p+1 (d i ) = c i 2 p−1 (see e.g. Lemma 2.5 in [9]). Moreover, its pth derivative is Lipschitz continuous with constant L p (d i ) = L p (f i ) + c i • p!…”

Section: General Regularization Schemementioning

confidence: 99%

Optimization Methods for Fully Composite Problems

Doikov,

Nesterov

2021

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper, we propose a new Fully Composite Formulation of convex optimization problems. It includes, as a particular case, the problems with functional constraints, max-type minimization problems, and problems of Composite Minimization, where the objective can have simple nondifferentiable components. We treat all these formulations in a unified way, highlighting the existence of very natural optimization schemes of different order. We prove the global convergence rates for our methods under the most general conditions. Assuming that the upper-level component of our objective function is subhomogeneous, we develop efficient modification of the basic Fully Composite first-order and second-order Methods, and propose their accelerated variants.

show abstract

“…Note that, with respect to the operator m j=1 c j c T j , the function f is 1-strongly convex and its Hessian is 2-Lipschitz continuous (see e.g. [28,Ex. 1]).…”

Section: Numerical Experimentsmentioning

confidence: 99%

Greedy Quasi-Newton Methods with Explicit Superlinear Convergence

Rodomanov,

Nesterov

2020

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper, we study greedy variants of quasi-Newton methods. They are based on the updating formulas from a certain subclass of the Broyden family. In particular, this subclass includes the well-known DFP, BFGS and SR1 updates. However, in contrast to the classical quasi-Newton methods, which use the difference of successive iterates for updating the Hessian approximations, our methods apply basis vectors, greedily selected so as to maximize a certain measure of progress. For greedy quasi-Newton methods, we establish an explicit non-asymptotic bound on their rate of local superlinear convergence, which contains a contraction factor, depending on the square of the iteration counter. We also show that these methods produce Hessian approximations whose deviation from the exact Hessians linearly convergences to zero.

show abstract

Minimizing Uniformly Convex Functions by Cubic Regularization of Newton Method

Cited by 16 publications

References 15 publications

Lower Complexity Bounds for Minimizing Regularized Functions

Lower Complexity Bounds for Minimizing Regularized Functions

Optimization Methods for Fully Composite Problems

Greedy Quasi-Newton Methods with Explicit Superlinear Convergence

Contact Info

Product

Resources

About