Expectation propagation in linear regression models with spike-and-slab priors

Abstract

An expectation propagation (EP) algorithm is proposed for approximate inference in linear regression models with spike-and-slab priors. This EP method is applied to regression tasks in which the number of training instances is small and the number of dimensions of the feature space is large. The problems analyzed include the reconstruction of genetic networks, the recovery of sparse signals, the prediction of user sentiment from customer-written reviews and the analysis of biscuit dough constituents from NIR spectra. The proposed EP method outperforms in most of these tasks another EP method that ignores correlations in the posterior and a variational Bayes technique for approximate inference. Additionally, the solutions generated by EP are very close to those given by Gibbs sampling, which can be taken as the gold standard but can be much more computationally expensive. In the tasks analyzed, spikeand-slab priors generally outperform other sparsifying priors, such as Laplace, Student's t and horseshoe priors. The key to the improved predictions with respect to Laplace and Student's t priors is the superior selective shrinkage capacity of the spike-and-slab prior distribution.

Knowledge elicitation via sequential probabilistic inference for high-dimensional prediction

“…To achieve fast computation, we use deterministic posterior approximations. Expectation propagation (Minka 2001) is used to approximate the spike-and-slab prior (Hernández-Lobato et al 2015) and the feedback models, and variational Bayes (e.g., Bishop 2006, Chapter 10) is used to approximate the residual variance σ 2 . The form of the posterior approximation for the regression coefficients w is Gaussian.…”

“…Expectation propagation has been found to provide good estimates of uncertainty, which is important in experimental design (Seeger 2008;Hernández-Lobato et al 2013;Hernández-Lobato et al 2015). In evaluating the expected information gain for a large number of candidate features, running the approximation iterations to full convergence for each is too slow, however.…”

“…Each review is one data point, and each distinct word is a feature with the corresponding covariate value given by the number of appearances of the word in the review. Sparse linear regression models have been shown suitable for this task in previous studies, for instance, in Hernández-Lobato et al (2015).…”

Parameter estimation in spike and slab variational inference for blind image deconvolution

“…Notice that the spike-and-slab prior can be approximated as the mixture of two Gaussians, one very peaky (the spike) and another with very high variance (the slab) [21] but this is still a mixture of two continuous distributions. According to [22] spike-and-slab models are more effective than other sparse priors (Laplacian or Student-t priors, for instance) in enforcing sparsity. The degree of sparsity can also be directly adjusted by modifying the weight of the spike in the mixture.…”

“…The work by Titsias et al [11], proposes an alternative VB inference model to approximate the posterior distribution using a simple and efficient algorithm. Instead of using a unimodal variational distribution, the authors propose an alternative approximation that more accurately matches the combinatorial nature of the posterior distribution over the spike-and-slab weights (see also [22] for the use of Expectation Propagation for posterior approximation).…”