On the Bernstein–von Mises phenomenon for nonparametric Bayes procedures

Abstract

We continue the investigation of Bernstein-von Mises theorems for nonparametric Bayes procedures from [Ann. Statist. 41 (2013-2028. We introduce multiscale spaces on which nonparametric priors and posteriors are naturally defined, and prove Bernstein-von Mises theorems for a variety of priors in the setting of Gaussian nonparametric regression and in the i.i.d. sampling model. From these results we deduce several applications where posterior-based inference coincides with efficient frequentist procedures, including Donskerand Kolmogorov-Smirnov theorems for the random posterior cumulative distribution functions. We also show that multiscale posterior credible bands for the regression or density function are optimal frequentist confidence bands.1. Introduction. The Bernstein-von Mises (BvM) theorem constitutes a powerful and precise tool to study Bayes procedures from a frequentist point of view. It gives universal conditions on the prior under which the posterior distribution has the approximate shape of a normal distribution. The theorem is well understood in finite-dimensional models (see [30] and [35]), but involves some delicate conceptual and mathematical issues in the infinite-dimensional setting. There exists a Donsker-type BvM theorem for the cumulative distribution function based on Dirichlet process priors, see Lo [31], and this carries over to a variety of closely related nonparametric situations, including quantile inference and censoring models, where Bernsteinvon Mises results are available: see [8,9,21,26,27] and [22]. The proofs of these results rely on a direct analysis of the posterior distribution, which is explicitly given in these settings (and typically of Dirichlet form).

Bernstein--von Mises Theorems and Uncertainty Quantification for Linear Inverse Problems

“…To the best of our knowledge, examples of Gaussian priors that attain a nonparametric BvM theorem in the optimal function space are known in literature only in the SVD-based framework considered in [7,8,45], or in the 'nearly-diagonal' problem studied very recently by [41]. Applying our proof to a Gaussian prior defined via SVD would here recover the result of [45].…”

“…One way of tackling the problem is to start by examining the limit behaviour of the one-dimensional marginals f , ψ W 1 | M ε instead of the full posterior. This semiparametric approach was introduced for a direct problem where A = I in [7,8], where it is shown that (approximately) in the small noise limit…”

“…Instead one has to employ some metric for weak convergence of probability measures. Utilising a Wassersteintype metric [7,8] achieve weak convergence of the posterior distribution to a prior-independent infinitedimensional Gaussian distribution on a large enough function space. More recently similar techniques were used in the inverse setting [38], for the linear X-ray transform problem, obtaining a semiparametric BvM theorem relative to smooth functionals of the unknown, while [40] proved a nonparametric result for a non-linear problem arising in partial differential equations.…”

On the Brittleness of Bayesian Inference

“…Although consistency is primarily a frequentist notion, according to Blackwell and Dubins [10] and Diaconis and Freedman [17], consistency is equivalent to intersubjective agreement, which means that two Bayesians will ultimately have very close predictive distributions. Fortunately, not only are there mild conditions which guarantee consistency, but the posterior distributions can be shown to contract/concentrate at an exponential rate around the data-generating distribution (see [55] for rates of contraction of posterior distributions based on Gaussian process priors) and the Bernstein-von Mises theorem goes further in providing mild conditions under which the posterior is asymptotically normal [13,14]. The most famous of these are Doob [19], Le Cam and Schwartz [39], and Schwartz [50,Thm.…”

Bayesian regression with nonparametric heteroskedasticity

“…The conditions are general but difficult to verify. Deriving easier to verify sufficient conditions for the non-and semiparametric Bernstein-von Mises theorem is an active area of current research, see, for example, Bickel and Kleijn (2012), Castillo (2012), Rivoirard and Rousseau (2012), Kleijn and Knapik (2012), Kato (2013), Castillo and Nickl (2013), Castillo and Rousseau (2013), and Castillo and Nickl (2014). Misspecified semiparametric models are not covered by the existing results.…”