Measures and algorithms for best basis selection

Abstract

A general framework based on majorization, Schur-concavity, and concavity is given that facilitates the analysis of algorithm performance and clarifies the relationships between existing proposed diversity measures useful for best basis selection. Admissible sparsity measures are given by the Schur-concave functions, which are the class of functions consistent with the partial ordering on vectors known as majorization. Concave functions form an important subclass of the Schur-concave functions which attain their minima at sparse solutions to the basis selection problem. Based on a particular functional factorization of the gradient, we give a general affine scaling optimization algorithm that converges to a sparse solution for measures chosen from within this subclass.

Performance tradeoffs for exact support recovery of sparse signals

“…2 For example, m = k log log k . 3 In this case, log m = ω(log k), and hence Ω(k log m) ⊂ Ω(k log k). Thus, n = Ω(k log k) is a better sufficient condition than n = Ω(k log m).…”

Dictionary Learning Algorithms for Sparse Representation

“…No formal proof of the convergence of this algorithm to the true maximum likelihood estimate, Â ML , has been given in the prior literature, but it appears to perform well in various test cases (Olshausen & Field, 1996). Below, we discuss the problem of dictionary learning within the framework of our recently developed log-prior model-based sparse source vector learning approach that for a known overcomplete dictionary can be used to obtain sparse codes (Rao, 1998;, 1998c, 1999. Such sparse codes can be found using FOCUSS, an affine scaling transformation (AST)-like iterative algorithm that finds a sparse locally optimal MAP estimate of the source vector x for an observation y.…”

“…However, we proceed by assuming the existence of a gradient factorization, (2.10) where α(x) is a positive scalar function and Π(x) is symmetric, positive-definite, and diagonal. As discussed in Kreutz-Delgado and Rao (1997Rao ( , 1998c and Rao and Kreutz-Delgado (1999), this assumption is generally true for CSC sparsity functions d p (·) and is key to understanding FOCUSS as a sparsity-inducing interior-point (AST-like) optimization algorithm. 5 With the gradient factorization 2.10, the stationary points of equation 2.9 are readily shown to be solutions to the (equally nonlinear and implicit) system, (2.11) (2.12) where β(x) = λα(x) and the second equation follows from identity A.18.…”

“…Taking β ≡ 0 in equation 2.13 yields the FOCUSS algorithm proved in Kreutz-Delgado and Rao (1997Rao ( , 1998c) and Rao and Kreutz-Delgado (1999) to converge to a sparse solution of equation 2.7 for CSC sparsity functions d p (·). The case β ≠ 0 yields the regularized FOCUSS algorithm that will converge to a sparse solution of equation 2.6 (Rao, 1998;Engan et al, 2000;Rao et al, 2002).…”

“…The sparsity of a vector is the number of zero-valued elements (Donoho, 1994), and is related to the diversity, the number of nonzero elements, Since our initial investigations into the properties of FOCUSS as an algorithm for providing sparse solutions to linear inverse problems in relatively noise-free environments (Gorodnitsky et al, 1995;Rao, 1997;Adler et al, 1996;, we now better understand the behavior of FOCUSS in noisy environments and as an interior point-like optimization algorithm for optimizing concave functionals subject to linear constraints , 1998c, 1999KreutzDelgado, Rao, Engan, Lee, & Sejnowski, 1999a;Engan, Rao, & Kreutz-Delgado, 2000;Rao, Engan, Cotter, & Kreutz-Delgado, 2002). In this article, we consider the use of the FOCUSS algorithm in the case where the matrix A is unknown and must be learned.…”