Bayesian l₀‐regularized least squares

Abstract

Bayesian l 0 -regularized least squares is a variable selection technique for high-dimensional predictors. The challenge is optimizing a nonconvex objective function via search over model space consisting of all possible predictor combinations. Spike-and-slab (aka Bernoulli-Gaussian) priors are the gold standard for Bayesian variable selection, with a caveat of computational speed and scalability. Single best replacement (SBR) provides a fast scalable alternative. We provide a link between Bayesian regularization and proximal updating, which provides an equivalence between finding a posterior mode and a posterior mean with a different regularization prior. This allows us to use SBR to find the spike-and-slab estimator. To illustrate our methodology, we provide simulation evidence and a real data example on the statistical properties and computational efficiency of SBR versus direct posterior sampling using spike-and-slab priors. Finally, we conclude with directions for future research. KEYWORDSBayesian variable selection, l 0 regularization, proximal updating, single best replacement, sparsity, spike-and-slab prior Appl Stochastic Models Bus Ind. 2019;35:717-731.wileyonlinelibrary.com/journal/asmb

Perfect reconstruction of sparse signals with piecewise continuous nonconvex penalties and nonconvexity control

“…The design of the nonconvex penalties that do not lead to a shrinkage of the basin is another possibility for nonconvex compressed sensing. From the relationship between the sparse prior in the Bayesian approach and the sparse penalty in the frequentist approach, it is implied that SCAD and MCP can be related to the Bernoulli-Gaussian prior in Bayesian terminology with large variance [25]. A unified understanding of the sparsity over the Bayesian and frequentist approach will be helpful for designing such desirable sparse penalties.…”

Ensembled best subset selection using summary statistics for polygenic risk prediction

“…L0L2 penalized regression allows for control over the number of selected SNPs (L0) and shrinkage of effect sizes to avoid overfitting (L2). These two penalties together can improve prediction performance by better leveraging SNPs in high LD compared to the L1 penalty used in Lassosum, and also have a conceptual connection to the Gaussian spike-and-slab prior commonly used in statistical genetics 13,15,33,[42][43][44][45][46] . Using a fast coordinate optimization algorithm, we can efficiently generate effect estimates, with a closed-form at each iteration, given a grid of L0 and L2 tuning parameter combinations.…”

Deep partial least squares for instrumental variable regression

“…discuss theoretical foundations of sparse rectified‐linear‐unit (ReLU) networks and the advantage of using Spike‐and‐Slab prior as an alternative to Dropout. They show that the resulting posterior prediction of ReLU networks with Spike‐and‐Slab regularization converge to a true function at a rate of $$$ lo{g}^{\delta }(n)/{n}^{-K} $$$ for $$$ \delta >1 $$$ and a positive constant $$$ K $$$ (with $$$ n $$$ number of observations) 34 . provide a theoretical connection between the Spike‐and‐Slab priors and $$$ {L}_0 $$$ norm regularization.…”

“…They show that the resulting posterior prediction of ReLU networks with Spike-and-Slab regularization converge to a true function at a rate of log 𝛿 (n)∕n −K for 𝛿 > 1 and a positive constant K (with n number of observations). 34 provide a theoretical connection between the Spike-and-Slab priors and L 0 norm regularization. They demonstrate that the regularized estimators can result in improved out-of-sample prediction performance.…”