Nested Expectation Propagation for Gaussian Process Classification with a Multinomial Probit Likelihood

Abstract

We consider probabilistic multinomial probit classification using Gaussian process (GP) priors. The challenges with the multiclass GP classification are the integration over the non-Gaussian posterior distribution, and the increase of the number of unknown latent variables as the number of target classes grows. Expectation propagation (EP) has proven to be a very accurate method for approximate inference but the existing EP approaches for the multinomial probit GP classification rely on numerical quadratures or independence assumptions between the latent values from different classes to facilitate the computations. In this paper, we propose a novel nested EP approach which does not require numerical quadratures, and approximates accurately all between-class posterior dependencies of the latent values, but still scales linearly in the number of classes. The predictive accuracy of the nested EP approach is compared to Laplace, variational Bayes, and Markov chain Monte Carlo (MCMC) approximations with various benchmark data sets. In the experiments nested EP was the most consistent method with respect to MCMC sampling, but the differences between the compared methods were small if only the classification accuracy is concerned.

A Tensor-Variate Gaussian Process for Classification of Multidimensional Structured Data

“…Our goal is to predict the class membership for a new input tensor X * given the observed data D = {X , y}. We place Gaussian process priors on the latent function related to each class, which is the common assumption in multiclass GP classification (see (Rasmussen and Williams 2006;Riihimäki, Jylänki, and Vehtari 2012)). This specification results in the following zeromean Gaussian prior for f :…”

“…Gaussian process can be extended to binary classification problems by employing logistic or probit likelihoods (Nickisch and Rasmussen 2008), while multinomial logistic or multinomial probit likelihoods are employed in multiclass Gaussian process classification (Williams and Barber 1998;Chai 2012;Girolami and Rogers 2006). Since exact inference is analytically intractable for logistic and probit likelihoods, approximation inference is widely applied, such as Laplace approximation (Williams and Barber 1998), expectation propagation (Kim and Ghahramani 2006;Riihimäki, Jylänki, and Vehtari 2012) and variational approximation (Girolami and Rogers 2006).…”