High-dimensional regression with gaussian mixtures and partially-latent response variables

Abstract

In this work we address the problem of approximating high-dimensional data with a lowdimensional representation. We make the following contributions. We propose an inverse regression method which exchanges the roles of input and response, such that the low-dimensional variable becomes the regressor, and which is tractable. We introduce a mixture of locally-linear probabilistic mapping model that starts with estimating the parameters of inverse regression, and follows with inferring closed-form solutions for the forward parameters of the high-dimensional regression problem of interest. Moreover, we introduce a partially-latent paradigm, such that the vector-valued response variable is composed of both observed and latent entries, thus being able to deal with data contaminated by experimental artifacts that cannot be explained with noise models. The proposed probabilistic formulation could be viewed as a latent-variable augmentation of regression. We devise expectation-maximization (EM) procedures based on a data augmentation strategy which facilitates the maximum-likelihood search over the model parameters. We propose two augmentation schemes and we describe in detail the associated EM inference procedures that may well be viewed as generalizations of a number of EM regression, dimension reduction, and factor analysis algorithms. The proposed framework is validated with both synthetic and real data. We provide experimental evidence that our method outperforms several existing regression techniques.

Switching Linear Inverse-Regression Model for Tracking Head Pose

“…In this section we summarize the mixture of linear inverse regressions of [8], which is named Gaussian locally linear mapping (GLLiM). GLLiM interchanges the roles of the input (high dimensional) and of the output (low dimensional), such that a low-to-high regression is being learned.…”

“…given the old parameter values θ (old) , while the maximization step computes new parameter values via maximization of the expected complete-data log-likelihood function, namely θ (new) = argmax E[log p(x, y, Z|θ (old) )], which yields a closed-form solution [8]. Initial responsibilities are obtained by fitting a K-component GMM to the lowdimensional data {x n } N n=1 .…”

“…Initial responsibilities are obtained by fitting a K-component GMM to the lowdimensional data {x n } N n=1 . Once the inverse parameter vector θ is estimated one obtains a low-dimensional to high-dimensional inverse predictive distribution [8]. The high-dimensional to lowdimensional forward predictive distribution has a closedform expression:…”

“…Not surprisingly, some of the best performing headpose estimation methods rely either on dimensionality reduction followed by regression, [35,32,16,19,4,12,36], or on high-dimensional-to-low-dimensional regression, e.g. [28,22,8,10].…”

“…Without loss of generality, we adopt a HOG-based description of faces, hence we need to predict a low-dimensional output (head-pose parameters) from a high-dimensional input (HOG vectors). To solve the latter we train the regression of [8] which is a generative Gaussian mixture of linear regression model. The proposed dynamic model is based on the switching Kalman filter (SKF) formulation.…”