Sparse optimization on measures with over-parameterized gradient descent

Abstract

Minimizing a convex function of a measure with a sparsity-inducing penalty is a typical problem arising, e.g., in sparse spikes deconvolution or two-layer neural networks training. We show that this problem can be solved by discretizing the measure and running nonconvex gradient descent on the positions and weights of the particles. For measures on a d-dimensional manifold and under some non-degeneracy assumptions, this leads to a global optimization algorithm with a complexity scaling as log(1/ ) in the desired accuracy , instead of −d for convex methods. The key theoretical tools are a local convergence analysis in Wasserstein space and an analysis of a perturbed mirror descent in the space of measures. Our bounds involve quantities that are exponential in d which is unavoidable under our assumptions.

SuperMix: Sparse regularization for mixtures

“…where F N is defined in Equation (47). As already emphasized in [11], Equation ( 15) is a convex program in µ whereas the parametrization given in H translates this minimization into a nonconvex differentiable problem in terms of r and t. This function H can be seen as an instance of the BLASSO Equation ( 15) for the measure µ a,t , namely a convex program that does not depends on the number of Particles N . All the more, it is possible to run a gradient descent on positions t i ∈ R d and weights r i > 0 of the N particles system.…”

“…Hence, for a large enough number of particles N , it then implies the convergence towards the global minimizer of µ a,t −→ F (µ a,t ) itself, despite the lack of convexity of the function (r, t) −→ F N (r, t). We refer to Theorem 3.9 of [11] that establishes the convergence of the particle gradient descent with a constant step-size under some non-degeneracy assumptions, i.e. the convergence of the CPGD towardμ n (15) in Hellinger-Kantorovich metric, and hence in the weak-sense.…”

“…Conic Particle Gradient Descent (CPGD) Conic Particle Gradient Descent [11] is an alternative promising avenue for solving BLASSO for Mixture Models (15). The idea is still to discretize a positive measure into a system of particles, i.e.…”

“…a sum of N Dirac masses following (47) with a i = r 2 i and use a mean-field approximation in the Wasserstein space jointly associated with a Riemannian gradient descent with the conic metric. We refer to [11] and the references therein for further details. This method may be shown to be rapid, with a log( −1 ) cost instead of −1/2 for standard convex programs.…”

Off-The-Grid Variational Sparse Spike Recovery: Methods and Algorithms

“…The following classes of algorithms better account for the infinite dimensional nature of M(X ). We present in detail the three methods with the most established results in the literature [13,18,23]. Before describing these methods, let us remark that there also exist some promising avenues, such as the projected gradient descent [24,25].…”