Fourier PCA and robust tensor decomposition

We consider the problem of efficiently learning mixtures of a large number of spherical Gaussians, when the components of the mixture are well separated. In the most basic form of this problem, we are given samples from a uniform mixture of k standard spherical Gaussians with means µ 1 , . . . , µ k ∈ R d , and the goal is to estimate the means up to accuracy δ using poly(k, d, 1/δ) samples.In this work, we study the following question: what is the minimum separation needed between the means for solving this task? The best known algorithm due to Vempala and Wang [JCSS 2004] requires a separation of roughly min{k, d} 1/4 . On the other hand, Moitra and Valiant [FOCS 2010] showed that with separation o(1), exponentially many samples are required. We address the significant gap between these two bounds, by showing the following results.• We show that with separation o( √ log k), super-polynomially many samples are required. In fact, this holds even when the k means of the Gaussians are picked at random in d = O(log k) dimensions.• We show that with separation Ω( √ log k), poly(k, d, 1/δ) samples suffice. Notice that the bound on the separation is independent of δ. This result is based on a new and efficient "accuracy boosting" algorithm that takes as input coarse estimates of the true means and in time (and samples) poly(k, d, 1/δ) outputs estimates of the means up to arbitrarily good accuracy δ assuming the separation between the means is Ω(min{ √ log k, √ d}) (independently of δ). The idea of the algorithm is to iteratively solve a "diagonally dominant" system of non-linear equations.We also (1) present a computationally efficient algorithm in d = O(1) dimensions with only Ω( √ d) separation, and (2) extend our results to the case that components might have different weights and variances. These results together essentially characterize the optimal order of separation between components that is needed to learn a mixture of k spherical Gaussians with polynomial samples.

show abstract

“…For any point y ∈ S j , y − µ j 2 ≤ 3σ j ( √ d + log(ρ w ρ σ )), and the separation of the means satisfies (37). To prove (25) we get from (38),…”

Section: Separation Of Order √ Dmentioning

confidence: 99%

On Learning Mixtures of Well-Separated Gaussians

Regev

Vijayaraghavan²

2017

2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS)

View full text Add to dashboard Cite

show abstract

“…A problem which has a similar flavor to dictionary learning is Independent Component Analysis (ICA), which has been a rich history in signal processing and computer science [Com94,FJK96,GVX14]. Here, we are given Y = AX where each entry of the matrix X is independent, and there are polynomial time algorithms both in the under-complete [FJK96] and over-complete case [DLCC07,GVX14] that recover A provided each entry of X is non-Gaussian. However, these algorithms do not apply in our setting, since the entries in each column of X are not independent (the supports can be almost arbitrarily correlated because of the adversarial samples).…”

Section: Related Workmentioning

confidence: 99%

Towards Learning Sparsely Used Dictionaries with Arbitrary Supports

Awasthi

Vijayaraghavan²

2018

2018 IEEE 59th Annual Symposium on Foundations of Computer Science (FOCS)

View full text Add to dashboard Cite

Dictionary learning is a popular approach for inferring a hidden basis in which data has a sparse representation. There is a hidden dictionary or basis A which is an n × m matrix, with m > n typically (this is called the over-complete setting). Data generated from the dictionary is given by Y = AX where X is a matrix whose columns have supports chosen from a distribution over k-sparse vectors, and the non-zero values chosen from a symmetric distribution. Given Y , the goal is to recover A and X in polynomial time (in m, n). Existing algorithms give polynomial time guarantees for recovering incoherent dictionaries, under strong distributional assumptions both on the supports of the columns of X, and on the values of the non-zero entries. In this work, we study the following question: can we design efficient algorithms for recovering dictionaries when the supports of the columns of X are arbitrary?To address this question while circumventing the issue of non-identifiability, we study a natural semirandom model for dictionary learning. In this model, there are a large number of samples y = Ax with arbitrary k-sparse supports for x, along with a few samples where the sparse supports are chosen uniformly at random. While the presence of a few samples with random supports ensures identifiability, the support distribution can look almost arbitrary in aggregate. Hence, existing algorithmic techniques seem to break down as they make strong assumptions on the supports.Our main contribution is a new polynomial time algorithm for learning incoherent over-complete dictionaries that provably works under the semirandom model. Additionally the same algorithm provides polynomial time guarantees in new parameter regimes when the supports are fully random. Finally, as a by product of our techniques, we also identify a minimal set of conditions on the supports under which the dictionary can be (information theoretically) recovered from polynomially many samples for almost linear sparsity, i.e., k = O(n). IntroductionIn many machine learning applications, the first step towards understanding the structure of naturally occurring data such as images and speech signals is to find an appropriate basis in which the data is sparse. Such sparse representations lead to statistical efficiency and can often uncover semantic features associated with the data. For example images are often represented using the SIFT basis [Low99]. Instead of designing an appropriate basis by hand, the goal of dictionary learning is to algorithmically learn from data, the basis (also known as the dictionary) along with the data's sparse representation in the dictionary. This problem of dictionary learning or sparse coding was first formalized in the seminal work of Olshausen and Field [OF97], and has now become an integral approach in unsupervised learning for feature extraction and data modeling.Informal Theorem 1.4 (Polynomial Identifiability for Rademacher Value Distribution). Consider a dictionary A ∈ R n×m that is (k, δ = 1/polylog(m))-RIP property for sparsi...

show abstract

“…For example, video may be considered a third-order tensor, with the first two modes describing the x and y pixel coordinates of a single frame, and the third mode representing time variations. Tensors have been used to represent and explore relationships among data in diverse research areas and applications, including video processing [15], multiarray signal processing [17], independent component analysis [7], and others.…”

Section: A Tensor Decompositionsmentioning

confidence: 99%

“…To alleviate the deficiencies encountered using method (7), this paper proposes using a classification method based on the decomposition of the tensor of training reservoirs. To this end, let approximations of the orthogonal Tucker decompositions of the tensors X and X k be…”

Section: Tensor Decompositions Of Reservoir Statesmentioning

confidence: 99%

Classification via tensor decompositions of echo state networks

Prater

2017

2017 IEEE Symposium Series on Computational Intelligence (SSCI)

View full text Add to dashboard Cite

This work introduces a tensor-based method to perform supervised classification on spatiotemporal data processed in an echo state network. Typically when performing supervised classification tasks on data processed in an echo state network, the entire collection of hidden layer node states from the training dataset is shaped into a matrix, allowing one to use standard linear algebra techniques to train the output layer. However, the collection of hidden layer states is multidimensional in nature, and representing it as a matrix may lead to undesirable numerical conditions or loss of spatial and temporal correlations in the data.This work proposes a tensor-based supervised classification method on echo state network data that preserves and exploits the multidimensional nature of the hidden layer states. The method, which is based on orthogonal Tucker decompositions of tensors, is compared with the standard linear output weight approach in several numerical experiments on both synthetic and natural data. The results show that the tensor-based approach tends to outperform the standard approach in terms of classification accuracy.

show abstract

Fourier PCA and robust tensor decomposition

Cited by 48 publications

References 66 publications

On Learning Mixtures of Well-Separated Gaussians

On Learning Mixtures of Well-Separated Gaussians

Towards Learning Sparsely Used Dictionaries with Arbitrary Supports

Classification via tensor decompositions of echo state networks

Contact Info

Product

Resources

About