Sparse Gaussian Process Variational Autoencoders

Abstract

Large, multi-dimensional spatio-temporal datasets are omnipresent in modern science and engineering. An effective framework for handling such data are Gaussian process deep generative models (GP-DGMs), which employ GP priors over the latent variables of DGMs. Existing approaches for performing inference in GP-DGMs do not support sparse GP approximations based on inducing points, which are essential for the computational efficiency of GPs, nor do they handle missing data -a natural occurrence in many spatio-temporal datasets -in a principled manner. We address these shortcomings with the development of the sparse Gaussian process variational autoencoder (SGP-VAE), characterised by the use of partial inference networks for parameterising sparse GP approximations. Leveraging the benefits of amortised variational inference, the SGP-VAE enables inference in multi-output sparse GPs on previously unobserved data with no additional training. The SGP-VAE is evaluated in a variety of experiments where it outperforms alternative approaches including multi-output GPs and structured VAEs.

Scalable Multi-Task Gaussian Processes with Neural Embedding of Coregionalization

“…Tables 1-3 showcase the comparative results of different models on the three multi-task datasets, respectively. It is notable that the results of GP and GPAR-NL in Tables 1 and 2 are taken from [28], and the results of GP-VAE are taken from [39]. We have the following findings from the comparative results.…”

“…This comparative study introduces state-of-the-art LMCs as well as other MTGP competitors, including (i) the Gaussian processes autoregressive regression with nonlinear correlations (GPAR-NL) [28], which has been verified to be superior in comparison to previous MTGPs, for example, CoKriging [56], intrinstic coregionalisation model (ICM) [57], semiparametric latent factor model (SLFM) [12], collaborative multi-output GP (CGP) [26], convolved multi-output GP (CMOGP) [37], and GPRN [22]; (ii) the multi-output GPs with neural likelihoods [31], including NMOGP, NGPRN and SVLMC-DKL; and (iii) the sparse GP variational autoencoders (GP-VAE) [39] with partial inference networks [58]. Besides, the baselines GP and stochastic variational GP (SVGP) [36] are involved in the comparison.…”

“…We first study the methodological characteristics of the proposed NSVLMC on a toy case. This toy case has three tasks generated from the same four latent functions expressed respectively as y 1 (x) =0.5f 1 (x) − 0.4f 2 (x) + 0.6f 3 (x) + 0.6f 4 (x) + , y 2 (x) = − 0.3f 1 (x) + 0.43f 2 (x) − 0.5f 3 (x) + 0.1f 4 (x) + , y 3 (x) =1.5f 1 (x) + 0.3f 3 (x) + 0.6f 4 (x) + , (39) where the independent noise ∼ N ( |0, 0.04), and the four latent functions are…”

“…To this end, we usually leverage the idea from scalable GPs that have been recently compared and reviewed in [32,33]. The majority of scalable LMCs relies on the framework of sparse approximation [34,35,36], which introduces M inducing variables to be the sufficient statistics of N latent function values for a task with M N , thus greatly reducing the cubic model complexity [37,26,38,39,40]. Other complexity reduction strategies have also been investigated through for example the distributed learning [41], the natural gradient assisted stochastic variational inference [42], and the exploitation of Kronecker structure in kernel matrix [43,44].…”

On Disentanglement in Gaussian Process Variational Autoencoders

“…In contrast, we show the efficacy of our approach to dynamic sequential data with dense changes of the factors in time. This general class of models has been further extended by the Sparse GP-VAE (SGP-VAE) [Ashman et al, 2020], the Scalable GP-VAE (SVGP-VAE) [Jazbec et al, 2020], and the Factorized GP-VAE (FGP-VAE) [Jazbec et al, 2021], which all improve their scalability to larger data sets. These extensions could also be readily applied to our model, which however we have not done in this study, since exact inference was still feasible in our experiments.…”