Scalable Importance Tempering and Bayesian Variable Selection

Zanella, Giacomo; Roberts, Gareth O.

doi:10.1111/rssb.12316

Cited by 33 publications

(47 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Unfortunately both the Bayes factors and the marginal relevance assessment have difficulties that make them unsatisfactory in our opinion. Firstly, the posterior inference via MCMC for multimodal posterior resulting from one of the sparsifying priors can be a challenge for high-dimensional feature spaces, albeit sophisticated sampling techniques can alleviate this problem (see, e.g., Zanella and Roberts, 2019). Secondly, for large number of features p the Bayes factors typically have high Monte Carlo errors due to the fact that only a vanishingly small proportion of the 2 p models is visited during MCMC, and almost all models are not visited at all.…”

Section: Bayes Factors and Marginal Posterior Relevance Assessmentmentioning

confidence: 99%

Projective inference in high-dimensional problems: Prediction and feature selection

Piironen¹,

Paasiniemi²,

Vehtari³

2020

Electron. J. Statist.

125

View full text Add to dashboard Cite

This paper reviews predictive inference and feature selection for generalized linear models with scarce but high-dimensional data. We demonstrate that in many cases one can benefit from a decision theoretically justified two-stage approach: first, construct a possibly non-sparse model that predicts well, and then find a minimal subset of features that characterize the predictions. The model built in the first step is referred to as the reference model and the operation during the latter step as predictive projection. The key characteristic of this approach is that it finds an excellent tradeoff between sparsity and predictive accuracy, and the gain comes from utilizing all available information including prior and that coming from the left out features. We review several methods that follow this principle and provide novel methodological contributions. We present a new projection technique that unifies two existing techniques and is both accurate and fast to compute. We also propose a way of evaluating the feature selection process using fast leave-one-out cross-validation that allows for easy and intuitive model size selection. Furthermore, we prove a theorem that helps to understand the conditions under which the projective approach could be beneficial. The key ideas are illustrated via several experiments using simulated and real world data.

show abstract

Section: Bayes Factors and Marginal Posterior Relevance Assessmentmentioning

confidence: 99%

Projective inference in high-dimensional problems: Prediction and feature selection

Piironen¹,

Paasiniemi²,

Vehtari³

2020

Electron. J. Statist.

125

View full text Add to dashboard Cite

show abstract

“…If said integrals can be obtained quickly, one can often use relatively simple algorithms to explore effectively the model space. For example, one may rely on the fast convergence of Metropolis–Hastings moves when posterior model probabilities concentrate (Yang et al., 2016), sequential Monte Carlo methods to lower the cost of model search (Schäfer & Chopin, 2013), tempering strategies to explore model spaces with strong multi‐modalities (Zanella & Roberts, 2019), or adaptive Markov Chain Monte Carlo to reduce the effort in exploring low posterior probability models (Griffin et al., 2020). Unfortunately, except for very specific settings such as Gaussian regression under conjugate priors, the integrated likelihood has no closed‐form, which seriously hampers scaling computations to even moderate dimensions.…”

Section: Figurementioning

confidence: 99%

Approximate Laplace Approximations for Scalable Model Selection

Rossell

Abril

Bhattacharya

2021

Journal of the Royal Statistical Society Series B: Statistical Methodology

View full text Add to dashboard Cite

show abstract

“…We note that if there is strong correlation between any two covariates, the coordinate‐wise updates, in Lines 2 to 5, might lead to slow mixing. In such situations, for example, the tempered Gibbs sampler as proposed in Zanella and Roberts (2019) can be applied. However, we note that it is not feasible to apply the tempered Gibbs sampler to all steps, because the density

p false(σ_{1}^{2} false| bold-italicβ, boldz, boldy, X false)

, in Line 7, cannot be evaluated analytically (see Section 5.4).…”

Section: Estimation Of Model Probabilitiesmentioning

confidence: 99%

Disjunct support spike‐and‐slab priors for variable selection in regression under quasi‐sparseness

Andrade

Fukumizu

2020

Stat

View full text Add to dashboard Cite

Sparseness of the regression coefficient vector is often a desirable property, because, among other benefits, sparseness improves interpretability. In practice, many true regression coefficients might be negligibly small, but nonzero, which we refer to as quasi‐sparseness. Spike‐and‐slab priors can be tuned to ignore very small regression coefficients and, as a consequence, provide a trade‐off between prediction accuracy and interpretability. However, spike‐and‐slab priors with full support lead to inconsistent Bayes factors, in the sense that the Bayes factors of any two models are bounded in probability. This is clearly an undesirable property for Bayesian hypotheses testing, where we wish that increasing sample sizes lead to increasing Bayes factors favoring the true model. As a remedy, we suggest disjunct support spike‐and‐slab priors, for which we prove consistent Bayes factors in the quasi‐sparse setting, and show experimentally fast growing Bayes factors favoring the true model. Several experiments on simulated and real data confirm the usefulness of our proposed method to identify models with high effect size, while leading to better control over false positives than hard‐thresholding.

show abstract

Scalable Importance Tempering and Bayesian Variable Selection

Cited by 33 publications

References 42 publications

Projective inference in high-dimensional problems: Prediction and feature selection

Projective inference in high-dimensional problems: Prediction and feature selection

Approximate Laplace Approximations for Scalable Model Selection

Disjunct support spike‐and‐slab priors for variable selection in regression under quasi‐sparseness

Contact Info

Product

Resources

About