Bayesian sparse linear regression with unknown symmetric error

Abstract

We study full Bayesian procedures for sparse linear regression when errors have a symmetric but otherwise unknown distribution. The unknown error distribution is endowed with a symmetrized Dirichlet process mixture of Gaussians. For the prior on regression coefficients, a mixture of point masses at zero and continuous distributions is considered. We study behavior of the posterior with diverging number of predictors. Conditions are provided for consistency in the mean Hellinger distance. The compatibility and restricted eigenvalue conditions yield the minimax convergence rate of the regression coefficients in ℓ 1 -and ℓ 2 -norms, respectively. The convergence rate is adaptive to both the unknown sparsity level and the unknown symmetric error density under compatibility conditions. In addition, strong model selection consistency and a semi-parametric Bernstein-von Mises theorem are proven under slightly stronger conditions.Keywords: Adaptive contraction rates, Bernstein von-Mises theorem, Dirichlet process mixture, high-dimensional semiparametric model, sparse prior, symmetric error * This article has been accepted for publication in Information and Inference Published by Oxford University Press. The accepted version contains significantly improved results, which is available at https://doi.org/10.1093/imaiai/iay022 the high-dimensional setting where p, the number of the predictors and the size of the coefficient vector, may grow with the sample size n, and possibly p ≫ n. If p > n, model (1) is not identifiable due to the singularity of its design matrix, therefore θ is not estimable unless further restrictions or structures are imposed. A standard assumption for θ is the sparsity condition which assumes that most components of θ are zero. For the last two decades, model (1) has been extensively studied under various sparsity conditions, in particular through penalized regression approaches such as Lasso and its various variants or extensions [36,37,46,47]. Recent advances in MCMC and other computational algorithms have led to a growing development of Bayesian models incorporating sparse priors [7,8,13,20,27]. In general, two classes of sparse priors are often used, the first being the spike and slab type (see e.g., [7,8]), with some recent work extending to continuous versions [20,28,32,33], and the other being continuous shrinkage priors; in particular, local-global shrinkage priors (see [1,5,31]).In the literature, both frequentist and Bayesian, the standard Gaussian error model, in which ǫ i 's are assumed to be i.i.d. Gaussian, is typically adopted, providing substantial computational and theoretical benefits. Using a squared error loss function, various penalization techniques are developed. Theoretical aspects of such estimates have been explored, showing recovery of θ in nearly optimal rate or optimal selection of the true nonzero coefficients [3,7,11,12,21]. More recent theoretical advances assure that relying on certain desparsifying techniques, asymptotically optimal (or at least honest) confidence set...

Unified Bayesian theory of sparse linear regression with nuisance parameters

“…For a Bernstein-von Mises theorem and selection consistency, stronger conditions are required than those used for posterior contraction, in line with the existing literature [e.g., 8,23]. As pointed out by Chae et al [10], the Bernsteinvon Mises theorems for finite dimensional parameters in classical semiparametric models [e.g., 7] may not be directly useful in the high-dimensional context. We thus directly characterize a version of the Bernstein von-Mises theorem for model (1).…”

“…In many semiparametric situations, however, it is often possible to obtain parametric rates for finite dimensional parameters under stronger conditions, even when there are infinite-dimensional nuisance parameters in a model [4,7]. It has also been shown that a similar argument holds in some high-dimensional semiparametric regression models [10]. Therefore, it is naturally of interest to examine under what conditions we can replace s by s 0 in the rates for θ, even if s 0 log p n¯ 2 n .…”

“…Extensive theoretical investigations, however, have been carried out only very recently. Since the pioneering work of Castillo et al [8], frequentist asymptotic properties of Bayesian sparse regression have been discovered under various settings, and there is now a substantial body of literature [e.g., 23,1,28,3,26,2,10,25,14,19,18]. Most of the existing studies deal with sparse regression setups without nuisance parameters and there are only a few exceptions.…”

“…In these studies, the optimal properties of Bayesian procedures are characterized with continuous shrinkage priors. For more involved models, Chae et al [10] adopted a nonparametric approach to estimate unknown symmetric densities in sparse linear regression. Ning et al [25] considered a sparse linear model for vectorvalued response variables with unknown covariance matrices.…”

Variational Bayes for High-Dimensional Linear Regression With Sparse Priors

“…This is one of the most widely used approaches within the Bayesian community (Mitchell and Beauchamp 1988;George and McCulloch 1993;West 2003;Efron 2008) and includes the popular spike-and-slab prior, which is often considered the gold standard in sparse Bayesian linear regression. Such priors have been shown to perform well for estimation and prediction (Johnstone and Silverman 2004;Castillo and van der Vaart 2012;Castillo, Schmidt-Hieber, and van der Vaart 2015;Chae, Lin, and Dunson 2019), uncertainty quantification (Ray 2017;Castillo and Szabó 2020), and multiple hypothesis testing (Castillo and Roquain 2020), see Banerjee, Castillo, and Ghosal (2020) for a recent review. relaxation is significant, reducing the posterior dimension to a much more tractable O(p).…”