When Gaussian Process Meets Big Data: A Review of Scalable GPs

Abstract

The vast quantity of information brought by big data as well as the evolving computer hardware encourages success stories in the machine learning community. In the meanwhile, it poses challenges for the Gaussian process (GP) regression, a well-known non-parametric and interpretable Bayesian model, which suffers from cubic complexity to data size. To improve the scalability while retaining desirable prediction quality, a variety of scalable GPs have been presented. But they have not yet been comprehensively reviewed and analyzed in order to be well understood by both academia and industry. The review of scalable GPs in the GP community is timely and important due to the explosion of data size. To this end, this paper is devoted to the review on state-of-the-art scalable GPs involving two main categories: global approximations which distillate the entire data and local approximations which divide the data for subspace learning. Particularly, for global approximations, we mainly focus on sparse approximations comprising prior approximations which modify the prior but perform exact inference, posterior approximations which retain exact prior but perform approximate inference, and structured sparse approximations which exploit specific structures in kernel matrix; for local approximations, we highlight the mixture/product of experts that conducts model averaging from multiple local experts to boost predictions. To present a complete review, recent advances for improving the scalability and capability of scalable GPs are reviewed. Finally, the extensions and open issues regarding the implementation of scalable GPs in various scenarios are reviewed and discussed to inspire novel ideas for future research avenues.Index Terms-Gaussian process regression, big data, scalability, sparse approximations, local approximations Haitao Liu is with the Rolls-

Generalized Convolution Spectral Mixture for Multi-Task Gaussian Processes

“…At present, efficient inference approximation methods like FITC and PITC [16,17,28,46,47,48,49,50], are not very effective for MTGPs, because in this setting, although inducing points can still be treated as free parameters, the task label should be pre-fixed. Interesting future research involves the development of sparse and efficient inference methods for MTGPs [19,51,52,53,54]. The initialization strategy [37,55,56,57] of the coupling hyperparamters C i is also very important for multitask pattern discovery, and needs further study.…”

Gaussian process classification for variable fidelity data

“…Finally, in order to make our method applicable to a wide range of real projects, its scalability should be improved. A number of approaches to do this has been recently reviewed [8,41].…”

An introduction of Gaussian processes and deep Gaussian processes and their applications to speech processing

“…Let N be the number of samples in training data. We require OðN 2 Þ storage for (N Â N) Gram matrix and OðN 3 Þ computational complexity for inversion and determinant in (16) and (11). This restricts N at most 10,000.…”