Oracle inequalities and optimal inference under group sparsity

Abstract

We consider the problem of estimating a sparse linear regression vector β * under a gaussian noise model, for the purpose of both prediction and model selection. We assume that prior knowledge is available on the sparsity pattern, namely the set of variables is partitioned into prescribed groups, only few of which are relevant in the estimation process. This group sparsity assumption suggests us to consider the Group Lasso method as a means to estimate β * . We establish oracle inequalities for the prediction and ℓ 2 estimation errors of this estimator. These bounds hold under a restricted eigenvalue condition on the design matrix. Under a stronger coherence condition, we derive bounds for the estimation error for mixed (2, p)-norms with 1 ≤ p ≤ ∞. When p = ∞, this result implies that a threshold version of the Group Lasso estimator selects the sparsity pattern of β * with high probability. Next, we prove that the rate of convergence of our upper bounds is optimal in a minimax sense, up to a logarithmic factor, for all estimators over a class of group sparse vectors. Furthermore, we establish lower bounds for the prediction and ℓ 2 estimation errors of the usual Lasso estimator. Using this result, we demonstrate that the Group Lasso can achieve an improvement in the prediction and estimation properties as compared to the Lasso.An important application of our results is provided by the problem of estimating multiple regression equation simultaneously or multi-task learning. In this case, our results lead to refinements of the results in [22] and allow one to establish the quantitative advantage of the Group Lasso over the usual Lasso in the multi-task setting. Finally, within the same setting, we show how our results can be extended to more general noise distributions, of which we only require the fourth moment to be finite. To obtain this extension, we establish a new maximal moment inequality, which may be of independent interest. 1 The phrase "β * is sparse" means that most of the components of this vector are equal to zero. 1 problem is relevant range from multi-task learning [2, 23, 28] and conjoint analysis [14,20] to longitudinal data analysis [11] as well as the analysis of panel data [15,38], among others. We briefly review these different settings in the course of the paper. In particular, multi-task learning provides a main motivation for our study. In that setting each regression equation corresponds to a different learning task; in addition to the requirement that M ≫ n, we also allow for the number of tasks T to be much larger than n. Following [2] we assume that there are only few common important variables which are shared by the tasks. That is, we assume that the vectors β * 1 , . . . , β * T are not only sparse but also have their sparsity patterns included in the same set of small cardinality. This group sparsity assumption induces a relationship between the responses and, as we shall see, can be used to improve estimation. The model (1.2) can be reformulated as a single regression problem of th...

Robust Inference on Average Treatment Effects with Possibly More Covariates than Observations

“…7 Our techniques build on prior studies, in particular Bickel, Ritov, and Tsybakov (2009), Lounici, Pontil, van de Geer, and Tsybakov (2011), Obozinski, Wainwright, and Jordan (2011), Belloni and Chernozhukov (2011), Belloni, Chen, Chernozhukov, and Hansen (2012), Belloni and Chernozhukov (2013), and Belloni, Chernozhukov, FernandezVal, and Hansen (2014).…”

Quantile regression with group lasso for classification

“…In these cases, the selection of groups of variables is of interest, rather than of individual variables. In order to address this type of problems, Yuan and Lin (2006) developed the group lasso method and a number of authors have subsequently extended it and studied its theoretical properties (Bach 2008;Huang and Zhang 2010;Wei and Huang 2010;Lounici et al 2011;Sharma et al 2013;Simon et al 2013). Given the merits of the regularized methods just described, regularized methods for binary response variables have also been developed.…”

Sparse multi-task reinforcement learning

“…Then we build on existing results in 107 multi-task regression [1,20] to improve the performance w.r.t. single-task learning.…”

“…Since from Lemma 1, s k t ≤ |J t | = s t for any iter- In order to exploit the similarity across tasks stated in Asm. 4, we resort to the Group LASSO (GL) algorithm [12,20], which defines a joint optimization problem over all the tasks. GL is based on the observation that, given the weight matrix W ∈ R d×T , the norm W 2,1 measures the level of shared-sparsity across tasks.…”