Lasso-type recovery of sparse representations for high-dimensional data

Abstract

The Lasso is an attractive technique for regularization and variable selection for high-dimensional data, where the number of predictor variables $p_n$ is potentially much larger than the number of samples $n$. However, it was recently discovered that the sparsity pattern of the Lasso estimator can only be asymptotically identical to the true sparsity pattern if the design matrix satisfies the so-called irrepresentable condition. The latter condition can easily be violated in the presence of highly correlated variables. Here we examine the behavior of the Lasso estimators if the irrepresentable condition is relaxed. Even though the Lasso cannot recover the correct sparsity pattern, we show that the estimator is still consistent in the $\ell_2$-norm sense for fixed designs under conditions on (a) the number $s_n$ of nonzero components of the vector $\beta_n$ and (b) the minimal singular values of design matrices that are induced by selecting small subsets of variables. Furthermore, a rate of convergence result is obtained on the $\ell_2$ error with an appropriate choice of the smoothing parameter. The rate is shown to be optimal under the condition of bounded maximal and minimal sparse eigenvalues. Our results imply that, with high probability, all important variables are selected. The set of selected variables is a meaningful reduction on the original set of variables. Finally, our results are illustrated with the detection of closely adjacent frequencies, a problem encountered in astrophysics.Comment: Published in at http://dx.doi.org/10.1214/07-AOS582 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org

Post-l1-penalized estimators in high-dimensional linear regression models

“…[2,6,10,13,20,19] demonstrated the fundamental result that ℓ 1 -penalized least squares estimators achieve the rate s/n √ log p, which is very close to the oracle rate s/n achievable when the true model is known. [17] demonstrated a similar fundamental result on the excess forecasting error loss under both quadratic and non-quadratic loss functions.…”

“…Several papers have begun to investigate estimation of HDSMs, primarily focusing on penalized mean regression, with the ℓ 1 -norm acting as a penalty function [2,6,10,13,17,20,19]. [2,6,10,13,20,19] demonstrated the fundamental result that ℓ 1 -penalized least squares estimators achieve the rate s/n √ log p, which is very close to the oracle rate s/n achievable when the true model is known.…”

“…In such models, the overall number of regressors p is very large, possibly much larger than the sample size n. However, the number s of significant regressorsthose having a non-zero impact on the response variable -is smaller than the sample size, that is, s = o(n). HDSMs ( [6], [13]) have emerged to deal with many new applications arising in biometrics, signal processing, machine learning, econometrics, and other areas of data analysis where high-dimensional data sets have become widely available.…”

“…In Section 2, we review some benchmark results of [2] for LASSO, albeit with a slightly improved choice of penalty, and model selection results of [11,13,21]. In Section 3, we present a generic rate result on post-penalized estimators.…”

Identifying small mean-reverting portfolios

“…A recent stream of works on the asymptotic consistency of these procedures can be found in Meinshausen & Yu (2007), Candes & Tao (2007), Banerjee et al (2007), Yuan & Lin (2007) or Rothman, Bickel, Levina & Zhu (2007) among others.…”