Forward–Backward Greedy Algorithms for Atomic Norm Regularization

Abstract

Abstract-In many signal processing applications, the aim is to reconstruct a signal that has a simple representation with respect to a certain basis or frame. Fundamental elements of the basis known as "atoms" allow us to define "atomic norms" that can be used to formulate convex regularizations for the reconstruction problem. Efficient algorithms are available to solve these formulations in certain special cases, but an approach that works well for general atomic norms, both in terms of speed and reconstruction accuracy, remains to be found. This paper describes an optimization algorithm called CoGEnT that produces solutions with succinct atomic representations for reconstruction problems, generally formulated with atomicnorm constraints. CoGEnT combines a greedy selection scheme based on the conditional gradient approach with a backward (or "truncation") step that exploits the quadratic nature of the objective to reduce the basis size. We establish convergence properties and validate the algorithm via extensive numerical experiments on a suite of signal processing applications. Our algorithm and analysis also allow for inexact forward steps and for occasional enhancements of the current representation to be performed. CoGEnT can outperform the basic conditional gradient method, and indeed many methods that are tailored to specific applications, when the enhancement and truncation steps are defined appropriately. We also introduce several novel applications that are enabled by the atomic-norm framework, including tensor completion, moment problems in signal processing, and graph deconvolution.

Cut Pursuit: Fast Algorithms to Learn Piecewise Constant Functions on General Weighted Graphs

“…Regularizing with an atomic gauge thus favors solutions that are sparse combinations of atoms, which motivated the use of algorithms that exploit the sparsity of the solution computationally [33,59]. It is clear from previous definitions that Lovász extensions are atomic gauges.…”

The Alternating Descent Conditional Gradient Method for Sparse Inverse Problems

“…One example of particular interest that has been studied in the context of the CGM is matrix completion [25,38,20,49]. In this case, the (b) step reduces to computing the leading singular vectors of a sparse matrix.…”

“…Indeed, it is well known that any modification of the iterate that decreases the objective function will not hurt theoretical convergence rates [24]. Moreover, Rao et al [38] have proposed a version of the conditional gradient method, called CoGENT, for atomic norm problems that take advantage of many common structures that arise in inverse problems. The reduction described in our theoretical analysis makes it clear that our algorithm can be seen as an instance of CoGENT specialized to the case of measures and differentiable measurement models.…”

Deconvolution of Point Sources: A Sampling Theorem and Robustness Guarantees

“…Quantifying the discretization error incurred by solving`1-norm minimization on a fine grid instead of solving the continuous TV norm minimization problem, in the spirit of [31]. Developing algorithms for TV norm minimization on a continuous domain; see [8,10,56] for some recent work in this direction. Extending our proof techniques to analyze deconvolution in multiple dimensions.…”