On the computational complexity of high-dimensional Bayesian variable selection

Abstract

We study the computational complexity of Markov chain Monte Carlo (MCMC) methods for high-dimensional Bayesian linear regression under sparsity constraints. We first show that a Bayesian approach can achieve variable-selection consistency under relatively mild conditions on the design matrix. We then demonstrate that the statistical criterion of posterior concentration need not imply the computational desideratum of rapid mixing of the MCMC algorithm. By introducing a truncated sparsity prior for variable selection, we provide a set of conditions that guarantee both variable-selection consistency and rapid mixing of a particular Metropolis-Hastings algorithm. The mixing time is linear in the number of covariates up to a logarithmic factor. Our proof controls the spectral gap of the Markov chain by constructing a canonical path ensemble that is inspired by the steps taken by greedy algorithms for variable selection.

Scalable Importance Tempering and Bayesian Variable Selection

“…(), section , a scenario similar to scenario (b) and Yang et al . () a scenario analogous to scenario (c). We compare GS, TGS and WTGS on all three scenarios for a variety of values of n , p and SNR.…”

“…We set the prior probability h to 5=p, corresponding to a prior expected number of active regressors equal Scenarios analogous to these have been previously considered in the literature. For example, Titsias and Yau (2017), section 3.2.3, considered a scenario similar to scenario (a), Wang et al (2011), example 4, andHuang et al (2016), section 4.2, a scenario similar to scenario (b) and Yang et al (2016) a scenario analogous to scenario (c). We compare GS, TGS and WTGS on all three scenarios for a variety of values of n, p and SNR.…”

“…(), Belloni and Chernozhukov (), Yang et al . () and Roberts and Rosenthal ()), whereas importance sampling weight variances tend to grow (often exponentially) with dimension. In this paper we propose a natural way to combine the best of MCMC and importance sampling in a way that is robust in high dimensional contexts and ameliorates the slow mixing which plagues many Markov‐chain‐based schemes.…”

“…schemes which simulate from precisely the right target distribution with no need for subsequent importance sampling correction) have been far more successful. This is because MCMC methods are usually much more scalable to high dimensional situations (see for example Frieze et al (1994), Belloni and Chernozhukov (2009), Yang et al (2016) and Roberts and Rosenthal (2016)), whereas importance sampling weight variances tend to grow (often exponentially) with dimension. In this paper we propose a natural way to combine the best of MCMC and importance sampling in a way that is robust in high dimensional contexts and ameliorates the slow mixing which plagues many Markov-chain-based schemes.…”

Scalable Bayesian Variable Selection Using Nonlocal Prior Densities in Ultrahigh-dimensional Settings

“…This consistency follows from that the fact that the BASAD “penalty” is asymptotically equivalent to $$lfalse({\widehat{\beta }}_{\mathbf{k}}false)-c\left|\mathrm{boldk}\right|\log false(pfalse),$$where c is some constant. Yang et al (2016) and Castillo et al (2012) also considered a similar penalty term on the model space, which implies that the posterior probability for their procedures can be expressed in the same form as (11). When g = p 2 c , the marginal likelihood based on a g -prior is asymptotically equivalent to (11).…”