Linear convergence of accelerated conditional gradient algorithms in spaces of measures

Abstract

A class of generalized conditional gradient algorithms for the solution of optimization problem in spaces of Radon measures is presented. The method iteratively inserts additional Dirac-delta functions and optimizes the corresponding coefficients. Under general assumptions, a sub-linear O(1/k) rate in the objective functional is obtained, which is sharp in most cases. To improve efficiency, one can fully resolve the finite-dimensional subproblems occurring in each iteration of the method. We provide an analysis for the resulting procedure: under a structural assumption on the optimal solution, a linear O (\zeta^k) convergence rate is obtained locally.

Asymptotic linear convergence of fully-corrective generalized conditional gradient methods

“…Moreover, if we interpret (1.3) as a sparse dictionary learning problem, in which the dictionary is given by Ext(B), they can be also linked to a fully corrective greedy selection method [35], or an accelerated gradient boosting algorithm [40]. Combining these three observations and carefully extending the techniques and results of [34] to the present setting we are able to conclude the linear convergence of our algorithm.…”

“…In this case, F is required to be smooth on dom(F ) and ∇F : dom(F ) → Y is assumed to be Lipschitz continuous on compact subsets of dom(F ). For more details, we refer the interested reader to [39] and [34].…”

“…We finally remind that z is the solution to (3.17) for ϕ = y d − ȳ, ȳ = K ū and therefore by the characterization of K * it holds that z(0) = −K * ∇F (K ū). The following structural assumptions are made, see also [32,34]. Assumption 3.13.…”

“…It turns out (see Section 4.1) that problems (P M ) and (P M + ) are equivalent in the sense that every minimizer to (P M ) can be converted to a solution of (P M + ) (and vice versa). Subsequently, in Section 4.3, we propose an extension of the Primal-Dual-Active-Point method from [34] to compute the solution of (P M + ) and we describe it in Algorithm 2. Finally, using the results of Section 4.1 we show that Algorithm 2 and Algorithm 1 are equivalent, setting the ground to prove Theorem 3.4 and Theorem 3.8 by using convergence results for Algorithm 2.…”

“…Algorithm 2, applied to (1.3). Such methods are accelerated variants of the generalized conditional gradient method for minimization problems over spaces of Borel measures introduced in [14] and their linear convergence has been recently proven for measures supported on a compact subset of the euclidean space, see [25,34]. Moreover, if we interpret (1.3) as a sparse dictionary learning problem, in which the dictionary is given by Ext(B), they can be also linked to a fully corrective greedy selection method [35], or an accelerated gradient boosting algorithm [40].…”