Bayesian linear regression with sparse priors

Abstract

We study full Bayesian procedures for high-dimensional linear regression under sparsity constraints. The prior is a mixture of point masses at zero and continuous distributions. Under compatibility conditions on the design matrix, the posterior distribution is shown to contract at the optimal rate for recovery of the unknown sparse vector, and to give optimal prediction of the response vector. It is also shown to select the correct sparse model, or at least the coefficients that are significantly different from zero. The asymptotic shape of the posterior distribution is characterized and employed to the construction and study of credible sets for uncertainty quantification.

Fast Bayesian Factor Analysis via Automatic Rotations to Sparsity

“…Given each γ jk ∈ {0, 1} for whether β jk should be ignored, a particularly appealing spike-and-slab variant has been π(β jk | γ jk , λ 1 ) = (1 − γ jk )δ 0 (β jk ) + γ jk φ(β jk | λ 1 ), (1.2) where δ 0 (·) is the "spike distribution" (atom at zero) and φ(· | λ 1 ) is the absolutely continuous "slab distribution" with exponential tails or heavier, indexed by a hyper-parameter λ 1 . Coupled with a suitable prior on γ jk , the generative model (1.2) yields optimal rates of posterior concentration, both in linear regression (Castillo and van der Vaart, 2012;Castillo et al, 2015) and covariance matrix estimation (Pati et al, 2014). This "methodological ideal", despite being amenable to posterior simulation, poses serious computational challenges in high-dimensional data.…”

“…Exponentially decaying priors on dimensionality are essential for obtaining well-behaved posteriors in sparse situations (Castillo et al, 2015;Castillo and van der Vaart, 2012). In the context of factor analysis, such priors were deployed by Pati et al (2014) to obtain rate-optimal posterior concentration around any true covariance matrix in spectral norm, when the dimension G = G n can be much larger than the sample size n. The following connection between the priors used by Pati et al (2014) and the IBP prior is illuminating.…”

“…As in Lemma 2 of Castillo et al (2015), we deploy a change of variables β S 0 − β 0S 0 → b to obtain…”

“…The last inequality follows from observing that P(||b|| ≤ ε/ √ 2) is the probability that first s events of a Poisson process of intensity λ 1 occur before time ε/ √ 2 (Castillo et al (2015), proof of Lemma 2).…”

Sparse Bayesian representation in time–frequency domain

“…To find sparse representations under overcomplete systems, pursuit algorithms (Chen et al, 1998;Mallat and Zhang, 1993) search for sparse dictionary approximations. Especially, the basis pursuit (BP) by Chen et al (1998), which is related to the LASSO (Tibshirani, 1996) in regression problem, focuses on the sparsity in redundant packet dictionaries by a principle of decomposing a signal into an optimal superposition of dictionary elements having the smallest ℓ 1 -norm of coefficients among all such decompositions.…”