Sparse nonnegative solution of underdetermined linear equations by linear programming

Abstract

Consider an underdetermined system of linear equations y ‫؍‬ Ax with known y and d ؋ n matrix A. We seek the nonnegative x with the fewest nonzeros satisfying y ‫؍‬ Ax. In general, this problem is NP-hard. However, for many matrices A there is a threshold phenomenon: if the sparsest solution is sufficiently sparse, it can be found by linear programming. We explain this by the theory of convex polytopes. Let aj denote the jth column of A, 1 < j Յ n, let a0 ‫؍‬ 0 and P denote the convex hull of the aj. We say the polytope P is outwardly k-neighborly if every subset of k vertices not including 0 spans a face of P. We show that outward k-neighborliness is equivalent to the statement that, whenever y ‫؍‬ Ax has a nonnegative solution with at most k nonzeros, it is the nonnegative solution to y ‫؍‬ Ax having minimal sum. We also consider weak neighborliness, where the overwhelming majority of k-sets of a js not containing 0 span a face of P. This implies that most nonnegative vectors x with k nonzeros are uniquely recoverable from y ‫؍‬ Ax by linear programming. Numerous corollaries follow by invoking neighborliness results. For example, for most large n by 2n underdetermined systems having a solution with fewer nonzeros than roughly half the number of equations, the sparsest solution can be found by linear programming.neighborly polytopes ͉ cyclic polytopes ͉ combinatorial optimization ͉ convex hull of Gaussian samples ͉ positivity constraints in ill-posed problems

Incorporating known features into a total variation dictionary model for source separation

“…We remark here, the authors of [9] proved that the following linear programming problem gives the sparsest representation available by dyadic intervals:…”

Identifying small mean-reverting portfolios

“…Sparse estimation techniques in general and the ℓ 1 penalization approach we use here in particular have also received a lot of attention in various forms: variable selection using the LASSO (see Tibshirani (1996)), sparse signal representation using basis pursuit by Chen, Donoho & Saunders (2001), compressed sensing (see Donoho & Tanner (2005) and Candès & Tao (2005)) or covariance selection (see Banerjee, Ghaoui & d'Aspremont (2007)), to cite only a few examples. A recent stream of works on the asymptotic consistency of these procedures can be found in Meinshausen & Yu (2007), Candes & Tao (2007), Banerjee et al (2007), Yuan & Lin (2007) or Rothman, Bickel, Levina & Zhu (2007) among others.…”

Performance guarantees for sparse regression-based unmixing

“…Previous attempts at predicting the performance of SRU algorithms [3] have relied on theoretical results linking the likelihood of obtaining useful sparse representations of mixed signals to the low degree of coherence (i.e., the largest normalized correlation) between the columns of the dictionary matrix and the high degree of sparseness of the abundance vectors (e.g., [7,8]). While coherence can be easily computed, it provides pessimistic guarantees in most cases due to the fact that such guarantees consider all sparse signals.…”