The predictive Lasso

Abstract

We propose a shrinkage procedure for simultaneous variable selection and estimation in generalized linear models (GLMs) with an explicit predictive motivation. The procedure estimates the coefficients by minimizing the Kullback-Leibler divergence of a set of predictive distributions to the corresponding predictive distributions for the full model, subject to an l1 constraint on the coefficient vector. This results in selection of a parsimonious model with similar predictive performance to the full model. Thanks to its similar form to the original lasso problem for GLMs, our procedure can benefit from available l1-regularization path algorithms. Simulation studies and real-data examples confirm the efficiency of our method in terms of predictive performance on future observations.

Penalized regression procedures for variable selection in the potential outcomes framework

“…By predictive criterion functions, we are referring to functions of predicted/imputed values of the response. Our general approach is to solve the following optimization problem [21]: $$\underset{\alpha }{\text{min}}\sum _{i=1}^n{D}_{i}false(M_{\mathit{\text{full}}},M_{\alpha }false)+\lambda \sum _{j=1}^{p}false|{\alpha }_{j}false|,$$ where λ > 0 is a tuning parameter and D i ( M full , M α ) refers to the Kullback-Leibler distance between the predictive distribution of a “full” model relative to those of a model parametrized by the vector α for subject i . When we talk about M here, we are actually alluding to models for the joint distribution of the potential outcomes.…”

“…The Kullback-Leibler distance is generically defined as $$Dfalse(f,gfalse)=true\int \text{log}true[\frac{gfalse(xfalse)}{ffalse(xfalse)}true]ffalse(xfalse)dx,$$ where f and g represent densities of the data. While the optimization problem in (7) is written down in a general form, what Tran et al [21] show is that in a linear model case, it corresponds to solving a weighted LASSO problem of the form minimizing $$\sum _{i=1}^n{false({\hat{\beta }}_0+{\hat{\beta }}_1{T}_i+\hat{\gamma }\prime {\mathbf{X}}_i-{\beta }_0-{\beta }_1{T}_i-\gamma \prime {\mathbf{X}}_{i}false)}^2\mathrm{normals}.\mathrm{normalt}.false|{\beta }_{0}false|+false|{\beta }_{1}false|+\sum _{j=1}^{p}false|{\gamma }_{j}false|\le t,$$ where (β̂ 0 , β̂ 1 ,γ̂) are the least squares estimators for the regression coefficients in the usual linear model for the observed data. Implicitly, we are assuming in (8) that the objective function is being evaluated at future design points which are identical to the observed data points.…”

“…The future design points are also referred to a test set in statistical parlance, which is different from the training data on which the model is built. Tran et al [21] prove their result using a canonical hierarchical normal model with conjugate priors, and in fact the objective function in (8) arises from the posterior predictive distribution in the model. Further details on the model used in [21] are given in Appendix A.1.…”

“…Tran et al [21] prove their result using a canonical hierarchical normal model with conjugate priors, and in fact the objective function in (8) arises from the posterior predictive distribution in the model. Further details on the model used in [21] are given in Appendix A.1. Another way to reinterpret their algorithm is as follows:

Using a training dataset, fit a linear regression of Y on T and X .

Based on the model fitted in step 1., compute predicted/fitted values of Y using the test dataset.

Solve equation (8) using the test data.

Again, there is an implicit assumption that the training and test datsets will come from the same distribution.…”

“…We also describe the conceptual flaws in applying off-the-shelf variable selection procedures to attempt to perform proper causal inference. The concept of predictive LASSO, as discussed in [21], is described. Section 3 features adaptation of this idea to the causal inference problem.…”