Adaptive Bernstein–von Mises theorems in Gaussian white noise

Ray, Kolyan

doi:10.1214/16-aos1533

Cited by 46 publications

(70 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent advances include extensions to extensions to semiparametric cases (Murphy and Van der Vaart, 2000;Kim et al, 2006;De Blasi and Hjort, 2009;Rivoirard et al, 2012;Castillo, 2012b,a;Castillo et al, 2014b;Panov and Spokoiny, 2014;Castillo et al, 2015;Ghosal and van der Vaart, 2017) and nonparametric cases (Cox, 1993;Freedman, 1986, 1997;Diaconis et al, 1998;Freedman et al, 1999;Kim and Lee, 2004;Boucheron et al, 2009;James et al, 2008;Johnstone, 2010;Bontemps et al, 2011;Kim, 2009;Knapik et al, 2011;Leahu et al, 2011;Rivoirard et al, 2012;Castillo and Nickl, 2012;Castillo et al, 2013;Spokoiny, 2013;Castillo et al, 2014a,b;Ray, 2014;Panov et al, 2015;Lu, 2017). In particular, proved a Bernstein-von Mises type result for Bayesian inverse problems, characterizing Gaussian approximations of probability measures with respect to the KL divergence.…”

Section: Definitionmentioning

confidence: 99%

Frequentist Consistency of Variational Bayes

Wang

Blei

2018

Journal of the American Statistical Association

139

117

View full text Add to dashboard Cite

A key challenge for modern Bayesian statistics is how to perform scalable inference of posterior distributions. To address this challenge, variational Bayes (VB) methods have emerged as a popular alternative to the classical Markov chain Monte Carlo (MCMC) methods. VB methods tend to be faster while achieving comparable predictive performance. However, there are few theoretical results around VB. In this paper, we establish frequentist consistency and asymptotic normality of VB methods. Specifically, we connect VB methods to point estimates based on variational approximations, called frequentist variational approximations, and we use the connection to prove a variational Bernstein-von Mises theorem. The theorem leverages the theoretical characterizations of frequentist variational approximations to understand asymptotic properties of VB. In summary, we prove that (1) the VB posterior converges to the Kullback-Leibler (KL) minimizer of a normal distribution, centered at the truth and (2) the corresponding variational expectation of the parameter is consistent and asymptotically normal. As applications of the theorem, we derive asymptotic properties of VB posteriors in Bayesian mixture models, Bayesian generalized linear mixed models, and Bayesian stochastic block models. We conduct a simulation study to illustrate these theoretical results.

show abstract

Section: Definitionmentioning

confidence: 99%

Frequentist Consistency of Variational Bayes

Wang

Blei

2018

Journal of the American Statistical Association

139

117

View full text Add to dashboard Cite

show abstract

“…Adaptive L 2 -credible regions with adequate frequentist coverage are constructed using the empirical Bayes approach in [34] for the Gaussian white noise model and in [27] for the nonparametric regression model using smoothing splines. In the setting of the Gaussian white noise model, Ray [23] constructed adaptive L 2 -credible sets using a weak BvM theorem, and also adaptive L ∞ -credible band using a spike and slab prior.…”

Section: Introduction Consider the Nonparametric Regression Modelmentioning

confidence: 99%

Supremum norm posterior contraction and credible sets for nonparametric multivariate regression

Yoo¹,

Ghosal²

2016

Ann. Statist.

View full text Add to dashboard Cite

In the setting of nonparametric multivariate regression with unknown error variance σ 2 , we study asymptotic properties of a Bayesian method for estimating a regression function f and its mixed partial derivatives. We use a random series of tensor product of B-splines with normal basis coefficients as a prior for f , and σ is either estimated using the empirical Bayes approach or is endowed with a suitable prior in a hierarchical Bayes approach. We establish pointwise, L2 and L∞-posterior contraction rates for f and its mixed partial derivatives, and show that they coincide with the minimax rates. Our results cover even the anisotropic situation, where the true regression function may have different smoothness in different directions. Using the convergence bounds, we show that pointwise, L2 and L∞-credible sets for f and its mixed partial derivatives have guaranteed frequentist coverage with optimal size. New results on tensor products of B-splines are also obtained in the course. MSC 2010 subject classifications: Primary 62G08; secondary 62G05, 62G15, 62G20 1 imsart-aos ver. 2014/10/16 file: supcredible_rev.tex date: October 9, 2018 arXiv:1411.6716v3 [math.ST] 24 Sep 2015 2 W. W. YOO AND S. GHOSALpointwise, L 2 and L ∞ (supremum) distances. We assume that the true regression function f 0 belongs to an anisotropic Hölder space (see Definition 2.1 below), and the errors under the true distribution are sub-Gaussian.Posterior contraction rates for regression functions in the L 2 -norm are well studied, but results for the stronger L ∞ -norm are limited. Giné and Nickl [14] studied contraction rates in L r -metric, 1 ≤ r ≤ ∞, and obtained optimal rate using conjugacy for the Gaussian white noise model and a rate for density estimation based on a random wavelet series and Dirichlet process mixture using a testing approach. In the same context, Castillo [2] introduced techniques based on semiparametric Bernstein-von Misses (BvM) theorems to obtain optimal L ∞ -contraction rates. Hoffman et al. [17] derived adaptive optimal L ∞ -contraction rate for the white noise model and also for density estimation. Scricciolo [25] applied the techniques of [14] to obtain L ∞ -rates using Gaussian kernel mixtures prior for analytic true densities.De Jonge and van Zanten [9] used finite random series based on tensor products of B-splines to construct a prior for nonparametric regression and derived adaptive L 2 -contraction rate for the regression function in the isotropic case. A BvM theorem for the posterior of σ is treated in [10]. Shen and Ghosal [28,29] used tensor products of B-splines respectively for Bayesian multivariate density estimation and high dimensional density regression in the anisotropic case.Nonparametric confidence bands for an unknown function were considered by [30, 1] and more recently by [6,13,5]. A Bayesian approaches the problem by constructing a credible set with a prescribed posterior probability. It is then natural to ask if the credible set has adequate frequentist coverage for large sample sizes. F...

show abstract

“…This is one of the most widely used approaches within the Bayesian community (Mitchell and Beauchamp 1988;George and McCulloch 1993;West 2003;Efron 2008) and includes the popular spike-and-slab prior, which is often considered the gold standard in sparse Bayesian linear regression. Such priors have been shown to perform well for estimation and prediction (Johnstone and Silverman 2004;Castillo and van der Vaart 2012;Castillo, Schmidt-Hieber, and van der Vaart 2015;Chae, Lin, and Dunson 2019), uncertainty quantification (Ray 2017;Castillo and Szabó 2020), and multiple hypothesis testing (Castillo and Roquain 2020), see Banerjee, Castillo, and Ghosal (2020) for a recent review. relaxation is significant, reducing the posterior dimension to a much more tractable O(p).…”

Section: Introductionmentioning

confidence: 99%

Variational Bayes for High-Dimensional Linear Regression With Sparse Priors

Ray

Szabó

2021

Journal of the American Statistical Association

Self Cite

View full text Add to dashboard Cite

We study a mean-field spike and slab variational Bayes (VB) approximation to Bayesian model selection priors in sparse high-dimensional linear regression. Under compatibility conditions on the design matrix, oracle inequalities are derived for the mean-field VB approximation, implying that it converges to the sparse truth at the optimal rate and gives optimal prediction of the response vector. The empirical performance of our algorithm is studied, showing that it works comparably well as other state-of-the-art Bayesian variable selection methods. We also numerically demonstrate that the widely used coordinate-ascent variational inference algorithm can be highly sensitive to the parameter updating order, leading to potentially poor performance. To mitigate this, we propose a novel prioritized updating scheme that uses a data-driven updating order and performs better in simulations. The variational algorithm is implemented in the R package sparsevb. Supplementary materials for this article are available online.

show abstract

Adaptive Bernstein–von Mises theorems in Gaussian white noise

Cited by 46 publications

References 48 publications

Frequentist Consistency of Variational Bayes

Frequentist Consistency of Variational Bayes

Supremum norm posterior contraction and credible sets for nonparametric multivariate regression

Variational Bayes for High-Dimensional Linear Regression With Sparse Priors

Contact Info

Product

Resources

About