Bayesian linear regression for multivariate responses under group sparsity

Abstract

We study frequentist properties of a Bayesian high-dimensional multivariate linear regression model with correlated responses. The predictors are separated into many groups and the group structure is pre-determined. Two features of the model are unique: (i) group sparsity is imposed on the predictors. (ii) the covariance matrix is unknown and its dimensions can also be high. We choose a product of independent spike-and-slab priors on the regression coefficients and a new prior on the covariance matrix based on its eigendecomposition. Each spike-and-slab prior is a mixture of a point mass at zero and a multivariate density involving a ℓ2,1-norm. We first obtain the posterior contraction rate, the bounds on the effective dimension of the model with high posterior probabilities. We then show that the multivariate regression coefficients can be recovered under certain compatibility conditions. Finally, we quantify the uncertainty for the regression coefficients with frequentist validity through a Bernstein-von Mises type theorem. The result leads to selection consistency for the Bayesian method. We derive the posterior contraction rate using the general theory by constructing a suitable test from the first principle using moment bounds for certain likelihood ratios. This leads to posterior concentration around the truth with respect to the average Rényi divergence of order 1/2. This technique of obtaining the required tests for posterior contraction rate could be useful in many other problems.

Empirical Priors and Coverage of Posterior Credible Sets in a Sparse Normal Mean Model

“…and Walker (2014), van der Pas et al (2017avan der Pas et al ( , 2014, and Ghosh and Chakrabarti (2015). Extensions to the high-dimensional regression setting can be found in Castillo et al (2015), Martin et al (2017), and Ning and Ghosal (2018).…”

Bayesian MIDAS Penalized Regressions: Estimation, Selection, and Prediction

“…Under a set of assumptions different from ours, and for a multivariate linear regression model with group sparsity, unknown covariance matrix, and spike-and-slab prior, Ning et al (2019) provide a posterior contraction rate for the in-sample prediction error similar to ours. Their prior structure differs from ours because they consider an independent prior, while our prior on the parameter of interest is conditional on the error variance parameter.…”

“…The contraction rate in Theorem 4.1 is the same as the one in Ning et al (2019) under our assumptions. The first term of the rate coincides (up to a logarithmic factor) with the minimax rate over a class of group sparse vectors derived in Lounici et al (2011).…”

“…Instead of relying on the general posterior contraction theory based on the Hellinger distance, which is not appropriate in this setting, the proof relies on the average Rényi divergence, which can be easily used to obtain the rate in terms of Euclidean distance. This is the approach followed in Ning et al (2019). To obtain the posterior contraction rate with respect to the Rényi divergence, we use the general posterior contraction procedure for independent observations, as in Ghosal and van der Vaart (2007), and we develop it for our particular framework.…”

“…As discussed in Bickel et al (2009), the parameter θ is not estimable without a condition on Z because of its large dimension. In particular, it is well known from the literature Castillo et al, 2015, andNing et al, 2019) that if θ is sparse, then local invertibility of the Gram matrix Z Z is sufficient for estimability of θ. We then introduce the following quantity.…”