Asymptotic linear convergence of fully-corrective generalized conditional gradient methods

Abstract

We propose an accelerated generalized conditional gradient method (AGCG) for the minimization of the sum of a smooth, convex loss function and a convex one-homogeneous regularizer over a Banach space. The algorithm relies on the mutual update of a finite set A k of extreme points of the unit ball of the regularizer and an iterate u k ∈ cone(A k ). Each iteration requires the solution of one linear problem to update A k and of one finite dimensional convex minimization problem to update the iterate. Under standard hypotheses on the minimization problem we show that the algorithm converges sublinearly to a solution. Subsequently, imposing additional assumptions on the associated dual variables, this is improved to a linear rate of convergence. The proof of both results relies on two key observations: First, we prove the equivalence of the considered problem to the minimization of a lifted functional over a particular space of Radon measures using Choquet's theorem. Second, the AGCG algorithm is connected to a Primal-Dual-Active-point Method (PDAP) on the lifted problem for which we finally derive the desired convergence rates.

Extremal points and sparse optimization for generalized Kantorovich-Rubinstein norms

“…The AGCG algorithm for non-smooth minimization, see [8] for the abstract algorithm in general Banach spaces, relies on the characterization of the extremal points and alternates between the update of a finite set of extremal points A k as well as of an iterate µ k in cone(A k ), the convex cone spanned by A k . Complexity-wise, every iteration of AGCG requires the solution of two subproblems: The minimization of a linear functional over Ext(B), to update A k , as well as the solution of a finite-dimensional, constrained minimization problem to improve the iterate µ k .…”

“…FISTA, interior point, or generalized Newton methods, we show that the former is equivalent to solving two finitedimensional, non-convex minimization problems. Moreover, based on the abstract results in [8], we present sufficient non-degeneracy conditions for the (fast) convergence of AGCG for (P).…”

“…It has been observed that an explicit description of such extremal points provides information on the structure of the sparse solutions of a regularized inverse problem when the observation is finite dimensional [5,4,28]. Moreover, it allows to devise accelerated generalized conditional gradient algorithms [8], i.e. infinite dimensional versions of the classical Frank-Wolfe algorithm [14,13,18,31] that are based on the iterative construction of linear combination of extremal points, converging to a solution of the minimization problem [20,7,16,6].…”

“…The main result of this paper, Theorem 2.5, gives a precise characterization of this set. Subsequently, we use these new-found extremal points to formulate simple first-order necessary and sufficient optimality conditions for (1) as well as to derive an efficient solution algorithm based on the accelerated generalized conditional gradient method presented in [8], see Algorithm 1 as well as Theorem 3.11 and 3.14, respectively.…”

Extremal points of total generalized variation balls in 1D: characterization and applications

“…Finally, note that evaluating TGV(u) already requires the solution of a nonsmooth minimization problem. Our method follows the spirit of [4] and alternates between the update of a set A k in Ext B via solving a linear minimization problem over B and improving the iterate u k ∈ cone(A k ) + L. The former can be implemented efficiently using Theorem 2 while the latter is done by minimizing a 1 -regularized surrogate functional over the finite dimensional set cone(A k ) + L. This completely avoids evaluating the TGV-functional throughout the iterations and eventually guarantees the subsequential convergence of {u k } k towards stationary points of (P).…”