Remarks on multi-output Gaussian process regression

Liu, Haitao; Cai, Jianfei; Ong, Yew-Soon

doi:10.1016/j.knosys.2017.12.034

Cited by 205 publications

(126 citation statements)

References 63 publications

Supporting

Mentioning

126

Contrasting

Order By: Relevance

“…The only difference is that unlike the zero-mean GP prior placed on f (x), we explicitly consider a prior mean µ 0 to account for the variability of the noise variance. 4 The kernels k f and k g could be, e.g., the squared exponential (SE) function equipped with automatic relevance determination (ARD) 3 The SVSHGP is implemented based on GPflow [51], which benefits from parallel/GPU speedup and automatic differentiation of Tensorflow [52]. The DVSHGP is built upon the GPML toolbox [53].…”

Section: A Sparse Approximationmentioning

confidence: 99%

See 1 more Smart Citation

Large-Scale Heteroscedastic Regression via Gaussian Process

Liu

Ong

Cai

2021

IEEE Trans. Neural Netw. Learning Syst.

Self Cite

View full text Add to dashboard Cite

Heteroscedastic regression which considers varying noises across input domain has many applications in fields like machine learning and statistics. Here we focus on the heteroscedastic Gaussian process (HGP) regression which integrates the latent function and the noise together in a unified nonparametric Bayesian framework. Though showing remarkable performance, HGP suffers from the cubic time complexity, which strictly limits its application to big data. To improve the scalability of HGP, we first develop a variational sparse inference algorithm, named VSHGP, to handle large-scale datasets. Furthermore, two variants are developed to further improve the scalability and capability of VSHGP. The first is stochastic VSHGP (SVSHGP) which derives a relaxed evidence lower bound factorized over data points, thus enhancing efficient stochastic variational inference. The second is distributed VSHGP (DVSHGP) which (i) follows the Bayesian committee machine formalism to distribute computations over multiple local VSHGP experts with many inducing points; and (ii) adopts hybrid parameters for experts to guard against over-fitting and capture local variety. Superiority of DVSHGP and SVSHGP as compared to existing scalable heteroscedastic/homoscedastic GPs is then verified using a synthetic dataset and four real-world datasets.

show abstract

Section: A Sparse Approximationmentioning

confidence: 99%

“…The DVSHGP is built upon the GPML toolbox [53]. 4 For f , we can pre-process the data to fulfill the zero-mean assumption. For g, however, it is hard to satisfy the zero-mean assumption, since we have no access to the "noise" data.…”

Section: A Sparse Approximationmentioning

confidence: 99%

Large-Scale Heteroscedastic Regression via Gaussian Process

Liu

Ong

Cai

2021

IEEE Trans. Neural Netw. Learning Syst.

Self Cite

View full text Add to dashboard Cite

show abstract

“…2) Multi-target Regression: The aim of multi-target regression is to simultaneously predict multiple real-valued output variables for one instance [5], [57]. Here, multiple labels are associated with each instance, represented by a real-valued vector, where the values represent how strongly the instance corresponds to a label.…”

Section: B Problem Definition Of Multi-output Learningmentioning

confidence: 99%

Survey on Multi-Output Learning

Shi

Tsang

et al. 2019

IEEE Trans. Neural Netw. Learning Syst.

Self Cite

161

View full text Add to dashboard Cite

The aim of multi-output learning is to simultaneously predict multiple outputs given an input. It is an important learning problem for decision-making, since making decisions in the real world often involves multiple complex factors and criteria. In recent times, an increasing number of research studies have focused on ways to predict multiple outputs at once. Such efforts have transpired in different forms according to the particular multi-output learning problem under study. Classic cases of multi-output learning include multi-label learning, multidimensional learning, multi-target regression and others. From our survey of the topic, we were struck by a lack in studies that generalize the different forms of multi-output learning into a common framework. This paper fills that gap with a comprehensive review and analysis of the multi-output learning paradigm. In particular, we characterize the 4 Vs of multi-output learning, i.e., volume, velocity, variety, and veracity, and the ways in which the 4 Vs both benefit and bring challenges to multioutput learning by taking inspiration from big data. We analyze the life cycle of output labeling, present the main mathematical definitions of multi-output learning, and examine the field's key challenges and corresponding solutions as found in the literature. Several model evaluation metrics and popular data repositories are also discussed. Last but not least, we highlight some emerging challenges with multi-output learning from the perspective of the 4 Vs as potential research directions worthy of further studies.

show abstract

“…Squared Exponential, Periodic, Matérn, and kernel design methods have been proposed [2]. The extension of GPs to multiple sources of data is known as multi-task Gaussian processes (MTGPs) [3]. MTGPs model temporal or spatial relationships among infinitely many random variables, as scalar GPs, but also account for the statistical dependence across different sources of data (or tasks) [3,4,5,6,7,8,9].…”

Section: Introductionmentioning

confidence: 99%

“…The extension of GPs to multiple sources of data is known as multi-task Gaussian processes (MTGPs) [3]. MTGPs model temporal or spatial relationships among infinitely many random variables, as scalar GPs, but also account for the statistical dependence across different sources of data (or tasks) [3,4,5,6,7,8,9]. How to choose an appropriate kernel to jointly model the cross covariance between tasks and auto-covariance within each task is the core aspect of MTGPs design [3,10,11,12,5,13,14].…”

Section: Introductionmentioning

confidence: 99%

Generalized Convolution Spectral Mixture for Multi-Task Gaussian Processes

Chen¹,

Laarhoven²,

Groot³

et al. 2018

Preprint

View full text Add to dashboard Cite

Multi-Task Gaussian processes (MTGPs) have shown a significant progress both in expressiveness and interpretation of the relatedness between different tasks: from linear combinations of independent single-output Gaussian processes (GPs), through the direct modeling of the cross-covariances such as spectral mixture kernels with phase shift, to the design of multivariate covariance functions based on spectral mixture kernels which model delays among tasks in addition to phase differences, and which provide a parametric interpretation of the relatedness across tasks. In this paper we further extend expressiveness and interpretability of MTGPs models and introduce a new family of kernels capable to model nonlinear correlations between tasks as well as dependencies between component, including time and phase delay. Specifically, we use generalized convolution spectral mixture kernels for modeling dependencies at component level, and coupling coregionalization for discovering task level correlations. The proposed kernels for MTGP are validated on artificial data and compared with existing MTGPs methods on three real-world experiments. Results indicate the benefits of our more expressive representation with respect to performance and interpretability.

show abstract

Remarks on multi-output Gaussian process regression

Cited by 205 publications

References 63 publications

Large-Scale Heteroscedastic Regression via Gaussian Process

Large-Scale Heteroscedastic Regression via Gaussian Process

Survey on Multi-Output Learning

Generalized Convolution Spectral Mixture for Multi-Task Gaussian Processes

Contact Info

Product

Resources

About