On the marginal likelihood and cross-validation

Fong, Edwin; Holmes, Chris

doi:10.48550/arxiv.1905.08737

Cited by 4 publications

(5 citation statements)

References 27 publications

(38 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As a consequence, the asymptotic results of this article apply equally to the case where σ is marginalised. Cross-validation offers more possibilities, both rooted (Fong and Holmes, 2019) and not rooted (e.g., Rathinavel and Hickernell, 2019, Section 2.2.3) in the GP model, to some of which our results on maximum likelihood may be relevant. An empirical investigation has been performed in Bachoc (2013).…”

Section: Discussionmentioning

confidence: 99%

Maximum likelihood estimation and uncertainty quantification for Gaussian process approximation of deterministic functions

Karvonen¹,

Wynne²,

Tronarp³

et al. 2020

Preprint

View full text Add to dashboard Cite

Despite the ubiquity of the Gaussian process regression model, few theoretical results are available that account for the fact that parameters of the covariance kernel typically need to be estimated from the dataset. This article provides one of the first theoretical analyses in the context of Gaussian process regression with a noiseless dataset. Specifically, we consider the scenario where the scale parameter of a Sobolev kernel (such as a Matérn kernel) is estimated by maximum likelihood. We show that the maximum likelihood estimation of the scale parameter alone provides significant adaptation against misspecification of the Gaussian process model in the sense that the model can become "slowly" overconfident at worst, regardless of the difference between the smoothness of the data-generating function and that expected by the model. The analysis is based on a combination of techniques from nonparametric regression and scattered data interpolation. Empirical results are provided in support of the theoretical findings.

show abstract

Section: Discussionmentioning

confidence: 99%

Maximum likelihood estimation and uncertainty quantification for Gaussian process approximation of deterministic functions

Karvonen¹,

Wynne²,

Tronarp³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…( 2), posterior belief distributions generated under the RoT are allowed to break a property referred to as coherence or Bayesian additivity (e.g. Bissiri et al, 2016;Fong and Holmes, 2019). In a nutshell, coherence says that posterior beliefs have to be generated according to some function ψ : R 2 → R which for the prior π(θ) and loss terms (θ, x 1 ), (θ, x 2 ) behaves as…”

Section: Coherence and The Rotmentioning

confidence: 99%

Generalized Variational Inference: Three arguments for deriving new Posteriors

Knoblauch,

Jewson,

Damoulas

2019

Preprint

View full text Add to dashboard Cite

In this paper we advocate an optimization-centric view on Bayesian statistics and introduce a novel generalization of Bayesian inference. On both counts, our inspiration is the representation of Bayes' rule as an infinite-dimensional optimization problem as shown independently by Csiszár (1975);Donsker and Varadhan (1975);Zellner (1988). First, we use this representation to prove a surprising optimality result of standard Variational Inference (VI) methods: Under the proposed view, the standard Evidence Lower Bound (ELBO) maximizing VI posterior is always preferable to alternative approximations of the Bayesian posterior. Next, we argue for an optimization-centric generalization of standard Bayesian inference. The need for this generalization arises in situations of severe misalignment between reality and three assumptions underlying the standard Bayesian posterior: (1) Well-specified priors, (2) well-specified likelihood models and (3) the availability of infinite computing power. In response to this observation, our generalization is defined by three arguments and named the Rule of Three (RoT). Each of its three arguments relaxes one of the assumptions underlying standard Bayesian inference. We axiomatically derive the RoT and recover existing methods as special cases, including the Bayesian posterior and its approximation by standard Variational Inference (VI). In contrast, alternative approximations to the Bayesian posterior maximizing other ELBO-like objectives violate these axioms. Finally, we introduce a special case of the RoT that we call Generalized Variational Inference (GVI).GVI posteriors are a large and tractable family of belief distributions specified by three arguments: A loss, a divergence and a variational family. GVI posteriors possess appealing theoretical properties, including consistency and an interpretation as an approximate ELBO.

show abstract

“…To increase stability, some have argued for repeated CV to optimize the penalty parameters (Boulesteix et al, 2017). A theoretical argument for repetition of subsampling is found in Fong and Holmes (2019), who establish an equivalence with marginal likelihood optimization.…”

Section: Extension 1: Repeated CVmentioning

confidence: 99%

Fast cross-validation for multi-penalty ridge regression

van de Wiel,

van Nee,

Rauschenberger

2020

Preprint

View full text Add to dashboard Cite

Prediction based on multiple high-dimensional data types needs to account for the potentially strong differences in predictive signal. Ridge regression is a simple, yet versatile and interpretable model for high-dimensional data that has challenged the predictive performance of many more complex models and learners, in particular in dense settings. Moreover, it allows using a specific penalty per data type to account for differences between those. Then, the largest challenge for multi-penalty ridge is to optimize these penalties efficiently in a cross-validation (CV) setting, in particular for GLM and Cox ridge regression, which require an additional loop for fitting the model by iterative weighted least squares (IWLS). Our main contribution is a computationally very efficient formula for the multi-penalty, sample-weighted hat-matrix, as used in the IWLS algorithm. As a result, nearly all computations are in the low-dimensional sample space. We show that our approach is several orders of magnitude faster than more naive ones. We developed a very flexible framework that includes prediction of several types of response, allows for unpenalized covariates, can optimize several performance criteria and implements repeated CV. Moreover, extensions to pair data types and to allow a preferential order of data types are included and illustrated on several cancer genomics survival prediction problems. The corresponding R-package, multiridge, serves as a versatile standalone tool, but also as a fast benchmark for other more complex models and multi-view learners.

show abstract

On the marginal likelihood and cross-validation

Cited by 4 publications

References 27 publications

Maximum likelihood estimation and uncertainty quantification for Gaussian process approximation of deterministic functions

Maximum likelihood estimation and uncertainty quantification for Gaussian process approximation of deterministic functions

Generalized Variational Inference: Three arguments for deriving new Posteriors

Fast cross-validation for multi-penalty ridge regression

Contact Info

Product

Resources

About