Local convergence of tensor methods

Doikov, Nikita; Nesterov, Yurii

doi:10.1007/s10107-020-01606-x

Cited by 14 publications

(11 citation statements)

References 18 publications

(21 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A similar convergence result has been obtained in [6] for a deterministic algorithm derived from the particular surrogate function of Example 3.2 with N = 1 (Taylor expansion with regularization). However, our convergence analysis of SHOM is derived for general surrogates.…”

Section: Local Linear Convergence In Function Valuessupporting

confidence: 75%

“…In particular, for strongly convex functions and p = 2 we recover the local linear rate from [11]. For the deterministic case, i.e the batch size is equal to the number of functions in the finite-sum, we obtain local superlinear convergence as in [6]. Numerical simulations also confirm the efficiency of our algorithm, i.e.…”

Section: Introductionsupporting

confidence: 58%

“…Choosing y = x * we obtain the right side of (13). On the other hand, from the uniform convexity relation (6) for the function f at x = x * , we have:…”

Section: Convex Local Convergence Analysismentioning

confidence: 99%

See 2 more Smart Citations

Convergence analysis of stochastic higher-order majorization-minimization algorithms

Lupu¹,

Necoara²

2021

Preprint

View full text Add to dashboard Cite

Majorization-minimization algorithms consist of successively minimizing a sequence of upper bounds of the objective function so that along the iterations the objective function decreases. Such a simple principle allows to solve a large class of optimization problems, even nonconvex, nonsmooth and stochastic. We present a stochastic higher-order algorithmic framework for minimizing the average of a very large number of sufficiently smooth functions. Our stochastic framework is based on the notion of stochastic higherorder upper bound approximations of the finite-sum objective function and minibatching. We present convergence guarantees for nonconvex and convex optimization when the higher-order upper bounds approximate the objective function up to an error that is p times differentiable and has a Lipschitz continuous p derivative. More precisely, we derive asymptotic stationary point guarantees for nonconvex problems, and for convex ones we establish local linear convergence results, provided that the objective function is uniformly convex. Unlike other higher-order methods, ours work with any batch size. Moreover, in contrast to most existing stochastic Newton and third-order methods, our approach guarantees local convergence faster than with first-order oracle and adapts to the problem's curvature. Numerical simulations also confirm the efficiency of our algorithmic framework.

show abstract

Section: Local Linear Convergence In Function Valuessupporting

confidence: 75%

Section: Introductionsupporting

confidence: 58%

See 1 more Smart Citation

Convergence analysis of stochastic higher-order majorization-minimization algorithms

Lupu¹,

Necoara²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…def = ∇g(T ) + φ (T ) ∈ ∂h(T ). In order to work with these objects, we use the following result (see Lemma 2 in [2]).…”

Section: This Inclusion Justifies Notation H (T )mentioning

confidence: 99%

Contracting Proximal Methods for Smooth Convex Optimization

Doikov,

Nesterov

2019

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper, we propose new accelerated methods for smooth Convex Optimization, called Contracting Proximal Methods. At every step of these methods, we need to minimize a contracted version of the objective function augmented by a regularization term in the form of Bregman divergence. We provide global convergence analysis for a general scheme admitting inexactness in solving the auxiliary subproblem. In the case of using for this purpose high-order Tensor Methods, we demonstrate an acceleration effect for both convex and uniformly convex composite objective function. Thus, our construction explains acceleration for methods of any order starting from one. The augmentation of the number of calls of oracle due to computing the contracted proximal steps, is limited by the logarithmic factor in the worst-case complexity bound.

show abstract

“…Subsequently, a high-order coordinate descent algorithm was studied in [9], and very recently, the high-order A-NPE framework has been specialized to the strongly convex setting [8], generalizing the discrete-time algorithms in this paper with an improved convergence rate. Beyond the setting of Lipschitz continuous derivatives, high-order algorithms and their accelerated variants have been adapted for more general setting with Hölder continuous derivatives [57,[63][64][65][66] and an optimal algorithm is known [105]. Other settings include structured convex non-smooth minimization [48], convex-concave minimax optimization and monotone variational inequalities [49,97], and structured smooth convex minimization [72,73,93,94].…”

Section: Introductionmentioning

confidence: 99%

A control-theoretic perspective on optimal high-order optimization

Lin

Jordan

2021

Math. Program.

View full text Add to dashboard Cite

We provide a control-theoretic perspective on optimal tensor algorithms for minimizing a convex function in a finite-dimensional Euclidean space. Given a function $$\varPhi : {\mathbb {R}}^d \rightarrow {\mathbb {R}}$$ Φ : R d → R that is convex and twice continuously differentiable, we study a closed-loop control system that is governed by the operators $$\nabla \varPhi $$ ∇ Φ and $$\nabla ^2 \varPhi $$ ∇ 2 Φ together with a feedback control law $$\lambda (\cdot )$$ λ ( · ) satisfying the algebraic equation $$(\lambda (t))^p\Vert \nabla \varPhi (x(t))\Vert ^{p-1} = \theta $$ ( λ ( t ) ) p ‖ ∇ Φ ( x ( t ) ) ‖ p - 1 = θ for some $$\theta \in (0, 1)$$ θ ∈ ( 0 , 1 ) . Our first contribution is to prove the existence and uniqueness of a local solution to this system via the Banach fixed-point theorem. We present a simple yet nontrivial Lyapunov function that allows us to establish the existence and uniqueness of a global solution under certain regularity conditions and analyze the convergence properties of trajectories. The rate of convergence is $$O(1/t^{(3p+1)/2})$$ O ( 1 / t ( 3 p + 1 ) / 2 ) in terms of objective function gap and $$O(1/t^{3p})$$ O ( 1 / t 3 p ) in terms of squared gradient norm. Our second contribution is to provide two algorithmic frameworks obtained from discretization of our continuous-time system, one of which generalizes the large-step A-HPE framework of Monteiro and Svaiter (SIAM J Optim 23(2):1092–1125, 2013) and the other of which leads to a new optimal p-th order tensor algorithm. While our discrete-time analysis can be seen as a simplification and generalization of Monteiro and Svaiter (2013), it is largely motivated by the aforementioned continuous-time analysis, demonstrating the fundamental role that the feedback control plays in optimal acceleration and the clear advantage that the continuous-time perspective brings to algorithmic design. A highlight of our analysis is that we show that all of the p-th order optimal tensor algorithms that we discuss minimize the squared gradient norm at a rate of $$O(k^{-3p})$$ O ( k - 3 p ) , which complements the recent analysis in Gasnikov et al. (in: COLT, PMLR, pp 1374–1391, 2019), Jiang et al. (in: COLT, PMLR, pp 1799–1801, 2019) and Bubeck et al. (in: COLT, PMLR, pp 492–507, 2019).

show abstract

Local convergence of tensor methods

Cited by 14 publications

References 18 publications

Convergence analysis of stochastic higher-order majorization-minimization algorithms

Convergence analysis of stochastic higher-order majorization-minimization algorithms

Contracting Proximal Methods for Smooth Convex Optimization

A control-theoretic perspective on optimal high-order optimization

Contact Info

Product

Resources

About