Tensor Decompositions for Learning Latent Variable Models

Abstract

This work considers a computationally and statistically efficient parameter estimation method for a wide class of latent variable models-including Gaussian mixture models, hidden Markov models, and latent Dirichlet allocation-which exploits a certain tensor structure in their loworder observable moments (typically, of second-and third-order). Specifically, parameter estimation is reduced to the problem of extracting a certain (orthogonal) decomposition of a symmetric tensor derived from the moments; this decomposition can be viewed as a natural generalization of the singular value decomposition for matrices. Although tensor decompositions are generally intractable to compute, the decomposition of these specially structured tensors can be efficiently obtained by a variety of approaches, including power iterations and maximization approaches (similar to the case of matrices). A detailed analysis of a robust tensor power method is provided, establishing an analogue of Wedin's perturbation theorem for the singular vectors of matrices. This implies a robust and computationally tractable estimation approach for several popular latent variable models.

A Provably Correct Learning Algorithm for Latent-Variable PCFGs

“…Recently a number of researchers have developed provably correct algorithms for parameter estimation in latent variable models such as hidden Markov models, topic models, directed graphical models with latent variables, and so on (Hsu et al, 2009;Bailly et al, 2010;Siddiqi et al, 2010;Parikh et al, 2011;Balle et al, 2011;Arora et al, 2013;Dhillon et al, 2012;Anandkumar et al, 2012;Arora et al, 2012;Arora et al, 2013). Many of these algorithms have their roots in spectral methods such as canonical correlation analysis (CCA) (Hotelling, 1936), or higher-order tensor decompositions.…”

On Generic Nonexistence of the Schmidt--Eckart--Young Decomposition for Complex Tensors

“…The definition of the SEY decomposition is not vacuous; it is shown in this paper that orthogonally diagonalizable tensors [18,58] satisfy the above conditions. Such tensors appear in several applications and have been extensively studied [3,6,14,18,46,58]; nevertheless, it does not appear to be known that this decomposition is optimal in the above sense. In fact, we prove that orthogonal diagonalizability is not a necessary condition for an SEY decomposition; a new class of optimally truncatable tensors is revealed in this paper.…”

Non-negative Spectral Learning for Linear Sequential Systems

“…In further works, we would like to investigate how using NNSpectral to initialize an EM algorithm and how adding constraints in NNSpectral to ensure the convergence of the series. In addition, relations with NMF could be further exploit to produce tensor [1] or kernel-based algorithms.…”

“…A detailed description of each problem can be found in [17]. Table 1 results among the following MoM-based algorithms : CO [6] using strings, Tensor [1] using strings and Spectral using strings and substrings. A description and comparison of these algorithms can be found in [5].…”

“…MoM-based algorithms are often consistent with theoretical guarantees in form of finite-sample bounds. In addition, these algorithms are able to learn a large variety of models [1], [3], [15], [11]. In a recent work, [15] showed that numerous models are encompassed into the common framework of linear Sequential Systems (SSs) or equivalently Multiplicity Automata (MA).…”