When are Overcomplete Representations Identifiable? Uniqueness of Tensor Decompositions Under Expansion Constraints

Anandkumar, Animashree; Hsu, Daniel; Janzamin, Majid; Kakade, Sham M.

doi:10.21236/ada604842

Cited by 23 publications

(39 citation statements)

References 40 publications

(87 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Similar (and yet not the same) expansion conditions have appeared in other contexts involving learning of overcomplete models. For instance, in [5], Anandkumar et. al.…”

Section: Discussionmentioning

confidence: 99%

A Clustering Approach to Learning Sparsely Used Overcomplete Dictionaries

Agarwal

Anandkumar

Netrapalli

2017

IEEE Trans. Inform. Theory

Self Cite

151

View full text Add to dashboard Cite

We consider the problem of learning overcomplete dictionaries in the context of sparse coding, where each sample selects a sparse subset of dictionary elements. Our main result is a strategy to approximately recover the unknown dictionary using an efficient algorithm. Our algorithm is a clustering-style procedure, where each cluster is used to estimate a dictionary element. The resulting solution can often be further cleaned up to obtain a high accuracy estimate, and we provide one simple scenario where ℓ 1 -regularized regression can be used for such a second stage.

show abstract

“…Similar (and yet not the same) expansion conditions have appeared in other contexts involving learning of overcomplete models. For instance, in [5], Anandkumar et. al.…”

Section: Discussionmentioning

confidence: 99%

A Clustering Approach to Learning Sparsely Used Overcomplete Dictionaries

Agarwal

Anandkumar

Netrapalli

2017

IEEE Trans. Inform. Theory

Self Cite

151

View full text Add to dashboard Cite

show abstract

“…As we know, Tucker models are not identifiable in general -there is linear transformation freedom. This can be alleviated when one can assume sparsity in C [132], G, or both (intuitively, this is because linear transformations generally do not preserve sparsity).…”

Section: F Topic Modelingmentioning

confidence: 99%

Tensor Decomposition for Signal Processing and Machine Learning

Sidiropoulos

Lathauwer

et al. 2017

IEEE Trans. Signal Process.

1,259

989

View full text Add to dashboard Cite

Tensors or {\em multi-way arrays} are functions of three or more indices $(i,j,k,\cdots)$ -- similar to matrices (two-way arrays), which are functions of two indices $(r,c)$ for (row,column). Tensors have a rich history, stretching over almost a century, and touching upon numerous disciplines; but they have only recently become ubiquitous in signal and data analytics at the confluence of signal processing, statistics, data mining and machine learning. This overview article aims to provide a good starting point for researchers and practitioners interested in learning about and working with tensors. As such, it focuses on fundamentals and motivation (using various application examples), aiming to strike an appropriate balance of breadth {\em and depth} that will enable someone having taken first graduate courses in matrix algebra and probability to get started doing research and/or developing tensor algorithms and software. Some background in applied optimization is useful but not strictly required. The material covered includes tensor rank and rank decomposition; basic tensor factorization models and their relationships and properties (including fairly good coverage of identifiability); broad coverage of algorithms ranging from alternating optimization to stochastic gradient; statistical performance analysis; and applications ranging from source separation to collaborative filtering, mixture and topic modeling, classification, and multilinear subspace learning.Comment: revised version, overview articl

show abstract

“…In addition, variants and generalizations of the problem (I.2) have also been studied in applications regarding control and optimization [25], nonrigid structure from motion [26], spectral estimation and Prony's problem [27], outlier rejection in PCA [28], blind source separation [29], graphical model learning [30], and sparse coding on manifolds [31]; see also [32] and the references therein.…”

Section: A Motivationmentioning

confidence: 99%

Finding a Sparse Vector in a Subspace: Linear Sparsity Using Alternating Directions

Sun

Wright

2016

IEEE Trans. Inform. Theory

117

View full text Add to dashboard Cite

Is it possible to find the sparsest vector (direction) in a generic subspace S ⊆ R p with dim (S) = n < p? This problem can be considered a homogeneous variant of the sparse recovery problem, and finds connections to sparse dictionary learning, sparse PCA, and many other problems in signal processing and machine learning. In this paper, we focus on a planted sparse model for the subspace: the target sparse vector is embedded in an otherwise random subspace. Simple convex heuristics for this planted recovery problem provably break down when the fraction of nonzero entries in the target sparse vector substantially exceeds O(1/ √ n). In contrast, we exhibit a relatively simple nonconvex approach based on alternating directions, which provably succeeds even when the fraction of nonzero entries is Ω(1). To the best of our knowledge, this is the first practical algorithm to achieve linear scaling under the planted sparse model. Empirically, our proposed algorithm also succeeds in more challenging data models, e.g., sparse dictionary learning.

show abstract

When are Overcomplete Representations Identifiable? Uniqueness of Tensor Decompositions Under Expansion Constraints

Cited by 23 publications

References 40 publications

A Clustering Approach to Learning Sparsely Used Overcomplete Dictionaries

A Clustering Approach to Learning Sparsely Used Overcomplete Dictionaries

Tensor Decomposition for Signal Processing and Machine Learning

Finding a Sparse Vector in a Subspace: Linear Sparsity Using Alternating Directions

Contact Info

Product

Resources

About