Bayesian sparse multiple regression for simultaneous rank reduction and variable selection

Abstract

We develop a Bayesian methodology aimed at simultaneously estimating low-rank and row-sparse matrices in a high-dimensional multiple-response linear regression model. We consider a carefully devised shrinkage prior on the matrix of regression coefficients which obviates the need to specify a prior on the rank, and shrinks the regression matrix towards low-rank and row-sparse structures. We provide theoretical support to the proposed methodology by proving minimax optimality of the posterior mean under the prediction risk in ultra-high dimensional settings where the number of predictors can grow sub-exponentially relative to the sample size. A one-step post-processing scheme induced by group lasso penalties on the rows of the estimated coefficient matrix is proposed for variable selection, with default choices of tuning parameters. We additionally provide an estimate of the rank using a novel optimization function achieving dimension reduction in the covariate space. We exhibit the performance of the proposed methodology in an extensive simulation study and a real data example.Short title: Bayesian sparse multi-task learner Lv = X T * ỹ , L T m = v, and L T w = z, where z ∼ N(0, I q 2 ). Finally obtain a sample as a = m + w.

Decoupling Shrinkage and Selection in Gaussian Linear Factor Analysis

“…However, one downside of our approach is the over shrinkage of the loadings values of the recovered sparse models. A possible solution to this problem would be the adoption of a particular case of our method, in which the posterior means of the loadings are used as weights as in R. P. Hahn and Carvalho (2015), Chakraborty et al (2020) and Woody et al (2019), which would mitigate the problem of over shrinkage.…”

“…The idea behind DSS is to explicit the trade-off between sparsity and the predictive performance of the model through a posterior summary obtained by sparse point-estimates generated from samples of the posterior distribution. In recent years, this framework has been extended in to a variety of statistical models (Puelz et al, 2017;Bashir et al, 2018;Woody et al, 2019;MacEachern and Miyawaki, 2019;Kowal and Bourgeois, 2020;Chakraborty et al, 2020).…”

Wheat Kernel Variety Identification Based on a Large Near-Infrared Spectral Dataset and a Novel Deep Learning-Based Feature Selection Method

“…The features with nonzero coefficients are considered as key features. As typical examples, the least absolute shrinkage and selection operator (LASSO) method (Kok et al, 2019;Chakraborty et al, 2020) is commonly used in regression problems. Logistic regression and linear SVM are adopted in classification problems (Pes, 2019).…”

Sparse time-varying parameter VECMs with an application to modeling electricity prices

“…First, we employ continuous globallocal shrinkage priors for pushing the parameter space towards sparsity. However, as noted by Chakraborty et al (2020), such priors solely achieve approximate zeroes, and the probability of observing exact zeroes is zero. As a remedy, we post-process our posterior draws via minimizing Lasso-type loss functions to obtain sparse estimates for the cointegration matrix, the autoregressive parameters and the covariance matrix (see also Hahn and Carvalho, 2015;Ray and Bhattacharya, 2018).…”

“…Both limitations are often impractical and identifying restrictions on the cointegration relationships can be tedious in large-scale models from an applied perspective. 2 Relevant contributions in this context are Bunea et al (2012), Jochmann et al (2013), Eisenstat et al (2016), Huber and Zörner (2019), Chakraborty et al (2020), Hauzenberger et al (2020a,c), Huber et al (2020b).…”