Affine-invariant contracting-point methods for Convex Optimization

Abstract

In this paper, we develop new affine-invariant algorithms for solving composite convex minimization problems with bounded domain. We present a general framework of Contracting-Point methods, which solve at each iteration an auxiliary subproblem restricting the smooth part of the objective function onto contraction of the initial domain. This framework provides us with a systematic way for developing optimization methods of different order, endowed with the global complexity bounds. We show that using an appropriate affine-invariant smoothness condition, it is possible to implement one iteration of the Contracting-Point method by one step of the pure tensor method of degree p ≥ 1. The resulting global rate of convergence in functional residual is then O(1/k p ), where k is the iteration counter. It is important that all constants in our bounds are affine-invariant. For p = 1, our scheme recovers well-known Frank-Wolfe algorithm, providing it with a new interpretation by a general perspective of tensor methods. Finally, within our framework, we present efficient implementation and total complexity analysis of the inexact second-order scheme (p = 2), called Contracting Newton method. It can be seen as a proper implementation of the trust-region idea. Preliminary numerical results confirm its good practical performance both in the number of iterations, and in computational time.

Analysis of the Frank–Wolfe method for convex composite optimization involving a logarithmically-homogeneous barrier

“…The Frank-Wolfe method is used therein to solve each projection sub-problem in the projected Newton method, and [40] shows that the total number of linear minimization sub-problems needed is O(ε −(1+o(1)) ). Another such example is in [17, Section 5], which develops an affine-invariant trust-region type of method for solving a class of convex composite optimization problems in a similar form as (1.1), with the key difference being that in [17] f is assumed to be twice differentiable with Lipschitz Hessian on dom h. The Frank-Wolfe method is used in [17] to solve each projection sub-problem, wherein it is shown that the total number of linear minimization sub-problems needed is O(ε −1 ).…”