A singular Woodbury and pseudo-determinant matrix identities and application to Gaussian process regression

A solution to the ill‐conditioning of gradient‐enhanced covariance matrices for Gaussian processes

“…A common approach to alleviate the ill‐conditioning of a matrix is to regularize it, that is, to add a positive nugget to its diagonal. For a GP, the addition of a nugget to the covariance matrix is analogous to having noisy data, 6,34 as can be seen between Equations (6) and (16) for the gradient‐free and gradient‐enhanced cases, respectively. When the nugget is zero, the mean of the posterior for the GP will match the function of interest exactly at all points where it has been evaluated.…”

“…The gradient‐free covariance matrix is commonly given by $$$$ \Sigma \left(\mathsf{X};{\hat{\sigma}}_{\mathsf{K}},\boldsymbol{\gamma}, {\hat{\sigma}}_0\right)={{\hat{\sigma}}_{\mathsf{K}}}^2\mathsf{K}\left(\mathsf{X};\boldsymbol{\gamma} \right)+{\hat{\sigma}}_0^2\mathsf{I}, $$$$ where the hyperparameter $$$ {{\hat{\sigma}}_{\mathsf{K}}}^2 $$$ is the variance of the stationary residual error and $$$ {\hat{\sigma}}_0^2 $$$ is a hyperparameter that estimates $$$ {\sigma}_0^2 $$$, which is the true noise variance 34 . The hyperparameter $$$ {\hat{\sigma}}_0^2 $$$ is used when the function evaluations are noisy and, in practice, it also serves to regularize $$$ \Sigma $$$ in order to reduce its condition number 34 . To separate the need to regularize the covariance matrix from the estimation of the uncertainty of the function evaluations, we use the following notation …”