Prediction with Gaussian Processes: From Linear Regression to Linear Prediction and Beyond

Abstract. We give a basic introduction to Gaussian Process regression models. We focus on understanding the role of the stochastic process and how it is used to define a distribution over functions. We present the simple equations for incorporating training data and examine how to learn the hyperparameters using the marginal likelihood. We explain the practical advantages of Gaussian Process and end with conclusions and a look at the current trends in GP work.Supervised learning in the form of regression (for continuous outputs) and classification (for discrete outputs) is an important constituent of statistics and machine learning, either for analysis of data sets, or as a subgoal of a more complex problem.Traditionally parametric 1 models have been used for this purpose. These have a possible advantage in ease of interpretability, but for complex data sets, simple parametric models may lack expressive power, and their more complex counterparts (such as feed forward neural networks) may not be easy to work with in practice. The advent of kernel machines, such as Support Vector Machines and Gaussian Processes has opened the possibility of flexible models which are practical to work with.In this short tutorial we present the basic idea on how Gaussian Process models can be used to formulate a Bayesian framework for regression. We will focus on understanding the stochastic process and how it is used in supervised learning. Secondly, we will discuss practical matters regarding the role of hyperparameters in the covariance function, the marginal likelihood and the automatic Occam's razor. For broader introductions to Gaussian processes, consult [1], [2]. Gaussian ProcessesIn this section we define Gaussian Processes and show how they can very naturally be used to define distributions over functions. In the following section we continue to show how this distribution is updated in the light of training examples.1 By a parametric model, we here mean a model which during training "absorbs" the information from the training data into the parameters; after training the data can be discarded.

show abstract

“…See e.g. [1] for the weight-space view of Gaussian processes which equivalently leads to Eq. (10) after marginalization over the weights.…”

Section: Training a Gaussian Processmentioning

confidence: 99%

“…Secondly, we will discuss practical matters regarding the role of hyperparameters in the covariance function, the marginal likelihood and the automatic Occam's razor. For broader introductions to Gaussian processes, consult [1], [2].…”

mentioning

confidence: 99%

Gaussian Processes in Machine Learning

Rasmussen

2004

Lecture Notes in Computer Science

4,885

5,889

View full text Add to dashboard Cite

show abstract

“…We use this method to deal with noise in our experiments. The derivation of (18) can be obtained by investigating the close relationship between Gaussian Processes (GP) and SVMs (Opper & Winther, 1999;Wahba, 1999;Williams, 1998). We give a brief description of it in the following for completeness.…”

Section: Input Noisementioning

confidence: 99%

Untitled

Long

2002

Machine Learning

132

View full text Add to dashboard Cite

Abstract. We describe a new incremental algorithm for training linear threshold functions: the Relaxed Online Maximum Margin Algorithm, or ROMMA. ROMMA can be viewed as an approximation to the algorithm that repeatedly chooses the hyperplane that classifies previously seen examples correctly with the maximum margin. It is known that such a maximum-margin hypothesis can be computed by minimizing the length of the weight vector subject to a number of linear constraints. ROMMA works by maintaining a relatively simple relaxation of these constraints that can be efficiently updated. We prove a mistake bound for ROMMA that is the same as that proved for the perceptron algorithm. Our analysis implies that the maximum-margin algorithm also satisfies this mistake bound; this is the first worst-case performance guarantee for this algorithm. We describe some experiments using ROMMA and a variant that updates its hypothesis more aggressively as batch algorithms to recognize handwritten digits. The computational complexity and simplicity of these algorithms is similar to that of perceptron algorithm, but their generalization is much better. We show that a batch algorithm based on aggressive ROMMA converges to the fixed threshold SVM hypothesis.

show abstract

“…For the moment, we do not specify the form of the covariance function and simply assume it is a valid one, generating a positive definite covariance matrix. We refer to [1,2,3,4] for a review of GPs.…”

Section: Introductionmentioning

confidence: 99%

Switching and Learning in Feedback Systems

Murray-Smith

Shorten²

2005

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract.With the Gaussian Process model, the predictive distribution of the output corresponding to a new given input is Gaussian. But if this input is uncertain or noisy, the predictive distribution becomes non-Gaussian. We present an analytical approach that consists of computing only the mean and variance of this new distribution (Gaussian approximation). We show how, depending on the form of the covariance function of the process, we can evaluate these moments exactly or approximately (within a Taylor approximation of the covariance function). We apply our results to the iterative multiple-step ahead prediction of non-linear dynamic systems with propagation of the uncertainty as we predict ahead in time. Finally, using numerical examples, we compare the Gaussian approximation to the numerical approximation of the true predictive distribution by simple Monte-Carlo.

show abstract

Prediction with Gaussian Processes: From Linear Regression to Linear Prediction and Beyond

Cited by 484 publications

References 26 publications

Gaussian Processes in Machine Learning

Gaussian Processes in Machine Learning

Untitled

Switching and Learning in Feedback Systems

Contact Info

Product

Resources

About