Asymptotic properties of the residual bootstrap for Lasso estimators

Abstract

Abstract. In this article, we derive the asymptotic distribution of the bootstrapped Lasso estimator of the regression parameter in a multiple linear regression model. It is shown that under some mild regularity conditions on the design vectors and the regularization parameter, the bootstrap approximation converges weakly to a random measure. The convergence result rigorously establishes a previously known heuristic formula for the limit distribution of the bootstrapped Lasso estimator. It is also shown that when one or more components of the regression parameter vector are zero, the bootstrap may fail to be consistent.

Bootstrapping Lasso Estimators

“…As seen in Chatterjee and Lahiri (2010), the inconsistency of the standard residual bootstrap arises when some components of β are zero. The key idea behind the modified bootstrap is to force components of the Lasso estimator β n to be exactly zero whenever they are close to zero.…”

“…For the bootstrap approximation to be useful, one would expect G n (·) to be close to G n (·). However, this is not the case; Chatterjee and Lahiri (2010) show that the residual bootstrap estimator G n (·), instead of converging to the deterministic limit of G n given by Knight and Fu (2000), converges weakly to a random probability measure and therefore, it fails to provide a valid approximation to G n (·). To appreciate why the residual bootstrap approximation have a random limit and why it is inconsistent, first observe that the Lasso estimators of the nonzero components of β estimate their signs correctly with Downloaded by [University of Connecticut] at 04:24 13 October 2014 high probability but the estimators of the zero components take both positive and negative values with positive probabilities, thereby erring to capture the target sign value (which is zero for such components) closely.…”

“…In contrast, for the standard version of the residual bootstrap, Chatterjee and Lahiri (2010), shows that under the same set of regularity assumptions, if λ 0 > 0 in Assumption (C.2) and if β has at least one zero component, then…”

“…Freedman 1981) for the Lasso estimator and sketched out its asymptotic behavior. Recently, it is further investigated rigorously by Chatterjee and Lahiri (2010), who show that the asymptotic distribution of the bootstrapped Lasso estimator is a random measure on R p and that the bootstrap is inconsistent whenever one or more components of the regression parameter is zero. Thus, in situations where the limit distribution of the Lasso estimator is most complicated and alternative approximations are needed the most, the usual bootstrap fails drastically!…”

“…Knight and Fu (2000) derived the asymptotic distribution of the Lasso estimator under Model (1.1) in the case where the dimension p of the regression is fixed. Properties of the standard bootstrap method have been investigated by Knight and Fu (2000) and Chatterjee and Lahiri (2010), in the same set up. In the finite dimensional case, Pötscher and coauthors have interesting results on the impossibility of estimating the distribution function of a Lasso estimator in a uniform sense.…”