A Unifying Review of Linear Gaussian Models

Abstract

Factor analysis, principal component analysis, mixtures of gaussian clusters, vector quantization, Kalman filter models, and hidden Markov models can all be unified as variations of unsupervised learning under a single basic generative model. This is achieved by collecting together disparate observations and derivations made by many previous authors and introducing a new way of linking discrete and continuous state models using a simple nonlinearity. Through the use of other nonlinearities, we show how independent component analysis is also a variation of the same basic generative model. We show that factor analysis and mixtures of gaussians can be implemented in autoencoder neural networks and learned using squared error plus the same regularization term. We introduce a new model for static data, known as sensible principal component analysis, as well as a novel concept of spatially adaptive observation noise. We also review some of the literature involving global and local mixtures of the basic models and provide pseudocode for inference and learning for all the basic models. A Unifying ReviewMany common statistical techniques for modeling multidimensional static data sets and multidimensional time series can be seen as variants of one underlying model. As we will show, these include factor analysis, principal component analysis (PCA), mixtures of gaussian clusters, vector quantization, independent component analysis models (ICA), Kalman filter models (also known as linear dynamical systems), and hidden Markov models (HMMs). The relationships between some of these models has been noted in passing in the recent literature. For example, Hinton, Revow, and Dayan (1995) note that FA and PCA are closely related, and Digalakis, Rohlicek, and Ostendorf (1993) relate the forward-backward algorithm for HMMs to * Present address: {roweis, zoubin}@gatsby.ucl.ac.uk.

A mixed filter algorithm for cognitive state estimation from simultaneously recorded continuous and binary measures of performance

“…Under the Gaussian approximation, the recursion defined by the probability densities in (5) and (6) becomes a recursive filter because it simplifies to computing recursively just the means and variances of these probability densities. In the special case that the state process is a linear Gaussian system and the observation model is a linear Gaussian function of the state process, this recursive computation of the means and variances is the Kalman filter (Fahrmeir and Tutz 2002;Kitagawa and Gersch 1996;Mendel 1995;Roweis and Ghahramani 1999). This would be true if our analysis did not include the binary performance measures.…”

“…State-space modeling is an established framework widely used to study dynamical processes in engineering, computer science and statistics (Fahrmeir and Tutz 2002;Kitagawa and Gersch 1996;Mendel 1995;Roweis and Ghahramani 1999). Applications of this paradigm range from control systems (Kitagawa and Gersch 1996;Mendel 1995), speech recognition (Becchetti and Ricotti 1999), studies of protein structure (White et al 1994), analysis of genetic networks (Harbison et al 2004), studies of neural representations of biological signals (Brown et al 1998;Barbieri et al 2004;Eden et al 2004;Smith and Brown 2003) and analyses of learning (Wirth et al 2003;Smith et al 2004).…”

“…The state equation defines an unobservable process whose evolution is tracked across time. Because the state process is unobservable, these models are often referred to as hidden Markov or latent process models (Fahrmeir and Tutz 2002;Roweis and Ghahramani 1999;Smith and Brown 2003;Smith et al 2004). The observation equation defines how the state is measured or recorded.…”

Distributed static linear Gaussian models using consensus

“…Roweis and Ghahramani (1999) showed how a single mathematical model, and a rather simple and commonly seen one at that, can be used to represent fully many different popular algorithms, such as PCA, FA, PPCA, Independent Component Analysis (ICA), Hidden Markov Models (HMM) and the Kalman filter, among others.…”

“…In each iteration, during the E-step, the factors (i.e. latent data) are obtained as the expectation X and covariance V of their posterior distribution given the observed data, using parameters of the last iteration (Roweis & Ghahramani, 1999).…”