Deep state-space Gaussian processes

Abstract

This paper is concerned with a state-space approach to deep Gaussian process (DGP) regression. We construct the DGP by hierarchically putting transformed Gaussian process (GP) priors on the length scales and magnitudes of the next level of Gaussian processes in the hierarchy. The idea of the state-space approach is to represent the DGP as a non-linear hierarchical system of linear stochastic differential equations (SDEs), where each SDE corresponds to a conditional GP. The DGP regression problem then becomes a state estimation problem, and we can estimate the state efficiently with sequential methods by using the Markov property of the state-space DGP. The computational complexity scales linearly with respect to the number of measurements. Based on this, we formulate state-space MAP as well as Bayesian filtering and smoothing solutions to the DGP regression problem. We demonstrate the performance of the proposed models and methods on synthetic non-stationary signals and apply the state-space DGP to detection of the gravitational waves from LIGO measurements.

Hierarchical Non-Stationary Temporal Gaussian Processes With $L^1$-Regularization

“…Recent methods and analysis of compositional NSGPs can be found in, for example, Wilson et al (2016a), Damianou andLawrence (2013), Calandra et al (2016), Al-Shedivat et al (2017), Wilson et al (2016b), andSalimbeni andDeisenroth (2017a). Hierarchically parameterized NSGPs have been recently studied, for example, by Zhao et al (2020), Emzir et al (2019), Monterrubio-Gómez et al (2020), Heinonen et al (2016), Roininen et al (2019), Cheng et al (2019), Dunlop et al (2018), Paciorek andSchervish (2006), andSchmidt andO'Hagan (2003). In particular, Heinonen et al (2016) and Zhao et al (2020) model the parameters of NSGPs as GPs, and they approximate the posterior distribution by using maximum a posteriori (MAP), Markov chain Monte Carlo (MCMC), and Bayesian smoothing methods.…”

“…Hierarchically parameterized NSGPs have been recently studied, for example, by Zhao et al (2020), Emzir et al (2019), Monterrubio-Gómez et al (2020), Heinonen et al (2016), Roininen et al (2019), Cheng et al (2019), Dunlop et al (2018), Paciorek andSchervish (2006), andSchmidt andO'Hagan (2003). In particular, Heinonen et al (2016) and Zhao et al (2020) model the parameters of NSGPs as GPs, and they approximate the posterior distribution by using maximum a posteriori (MAP), Markov chain Monte Carlo (MCMC), and Bayesian smoothing methods. Dunlop et al (2018) discuss the connections between the compositional and parameterized NSGPs.…”

“…Approximation methods for the posterior inference include, for example, variational Bayes and Markov chain Monte Carlo methods (Heinonen et al, 2016;Salimbeni and Deisenroth, 2017b). In the temporal case, Bayesian smoothing methods can be used (Zhao et al, 2020).…”

“…One approach is to use stochastic (partial) differential equation based constructions for this type of NS-GPs (Zhao et al, 2020;Emzir et al, 2020), which have the advantage that the covariance function is always valid by construction. In the case of temporal processes, one can convert NSGPs into state-space models which can be solved by Bayesian smoothers efficiently (Zhao et al, 2020).…”

“…The regularization is also closely related to the LASSO regression and elastic nets (Hastie et al, 2015). We consider two NSGP constructions which use explicitly constructed non-stationary covariance functions (Salimbeni and Deisenroth, 2017b;Heinonen et al, 2016) or stochastic differential equation (SDE) representations (Zhao et al, 2020), respectively.…”