Convex optimization based on global lower second-order models

Abstract

In this paper, we present new second-order algorithms for composite convex optimization, called Contracting-domain Newton methods. These algorithms are affine-invariant and based on global second-order lower approximation for the smooth component of the objective. Our approach has an interpretation both as a second-order generalization of the conditional gradient method, or as a variant of trust-region scheme. Under the assumption, that the problem domain is bounded, we prove O(1/k 2 ) global rate of convergence in functional residual, where k is the iteration counter, minimizing convex functions with Lipschitz continuous Hessian. This significantly improves the previously known bound O(1/k) for this type of algorithms. Additionally, we propose a stochastic extension of our method, and present computational results for solving empirical risk minimization problem.

Affine-invariant contracting-point methods for Convex Optimization

“…For p = 1, this recovers well-known result about global convergence of the classical Frank-Wolfe algorithm [10,19]. For p = 2, we obtain Contracting-Domain Newton Method from [8]. Thus, our analysis also extends the results from these works to the case, when the corresponding subproblem is solved inexactly.…”

“…Our framework extends the initial results presented in [19] and in [8]. In [19], it was shown that the classical Frank-Wolfe algorithm can be generalized onto the case of the composite objective function [18] using a contraction of the feasible set towards the current test point.…”

“…We see that the right hand side in ( 13) is the same, as that one in (8). However, this convergence is stronger, since it provides a bound for the accuracy certificate k .…”

“…For p = 1 and ψ(•) being an indicator function of a compact convex set, this is well-known Frank-Wolfe algorithm [10]. For p = 2, this is Contracting-Domain Newton Method from [8].…”

“…This operation was used there also for justifying a second-order method with contraction, which looks similar to the classical trust-region methods [5], but with asymmetric trust region. The convergence rates for the second-order methods with contractions were significantly improved in [8]. In this paper, we extend the contraction technique onto the whole family of tensor methods.…”