The degrees of freedom of the Lasso for general design matrix

Dossal, Charles; Kachour, Maher; Fadili, Jalal M.; Peyré, Gabriel; Chesneau, Christophe

doi:10.5705/ss.2011.281

Cited by 24 publications

(26 citation statements)

References 37 publications

(61 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Let further |I| = k and P I ∈ R k×n be a projector onto I and A I the restriction of A to I. We have that df α = x α (y) 0 = k and gdf α = tr(ΠB [J] ), B [J] := P I (A * I A I ) −1 P * I , as shown, e.g., in [39,14,12], which allows us to compute PSURE (7) and SURE (9). Notice that whilex α (y) is a continuous function of α [7], PSURE and SURE are discontinuous at all α where the support I changes.…”

Section: Numerical Studies For Non-quadratic Regularizationmentioning

confidence: 99%

Risk estimators for choosing regularization parameters in ill-posed problems - properties and limitations

Lucka

Proksch²,

Brüne³

et al. 2018

Inverse Problems &Amp; Imaging

View full text Add to dashboard Cite

This paper discusses the properties of certain risk estimators that recently regained popularity for choosing regularization parameters in ill-posed problems, in particular for sparsity regularization. They apply Stein's unbiased risk estimator (SURE) to estimate the risk in either the space of the unknown variables or in the data space, which we call PSURE in order to distinguish the two different risk functions. It seems intuitive that SURE is more appropriate for ill-posed problems, since the properties in the data space do not tell much about the quality of the reconstruction. We provide theoretical studies of both approaches for linear Tikhonov regularization in a finite dimensional setting and estimate the quality of the risk estimators, which also leads to asymptotic convergence results as the dimension of the problem tends to infinity. Unlike previous works which studied single realizations of image processing problems with a very low degree of ill-posedness, we are interested in the statistical behaviour of the risk estimators for increasing ill-posedness. Interestingly, our theoretical results indicate that the quality of the SURE risk can deteriorate asymptotically for ill-posed problems, which is confirmed by an extensive numerical study. The latter shows that in many cases the SURE estimator leads to extremely small regularization parameters, which obviously cannot stabilize the reconstruction. Similar but less severe issues with respect to robustness also appear for the PSURE estimator, which in comparison to the rather conservative discrepancy principle leads to the conclusion that regularization parameter choice based on unbiased risk estimation is not a reliable procedure for ill-posed problems. A similar numerical study for sparsity regularization demonstrates that the same issue appears in non-linear variational regularization approaches.

show abstract

Section: Numerical Studies For Non-quadratic Regularizationmentioning

confidence: 99%

Risk estimators for choosing regularization parameters in ill-posed problems - properties and limitations

Lucka

Proksch²,

Brüne³

et al. 2018

Inverse Problems &Amp; Imaging

View full text Add to dashboard Cite

show abstract

“…the lasso), the Jacobian matrix depends on the support (set of non-zero coefficients) of any lasso solution x(y, θ). An estimator of the DOF can then be retrieved from the number of non-zero entries of this solution [23,65,73]. These results have in turn been extended to more general sparsity promoting regularizations [20,40,61,[64][65][66]72], and spectral regularizations (e.g.…”

Section: Closed-form Surementioning

confidence: 99%

Stein Unbiased GrAdient estimator of the Risk (SUGAR) for Multiple Parameter Selection

Deledalle¹,

Vaiter²,

Fadili³

et al. 2014

SIAM J. Imaging Sci.

110

View full text Add to dashboard Cite

Algorithms to solve variational regularization of ill-posed inverse problems usually involve operators that depend on a collection of continuous parameters. When these operators enjoy some (local) regularity, these parameters can be selected using the socalled Stein Unbiased Risk Estimate (SURE). While this selection is usually performed by exhaustive search, we address in this work the problem of using the SURE to efficiently optimize for a collection of continuous parameters of the model. When considering non-smooth regularizers, such as the popular ℓ 1 -norm corresponding to soft-thresholding mapping, the SURE is a discontinuous function of the parameters preventing the use of gradient descent optimization techniques. Instead, we focus on an approximation of the SURE based on finite differences as proposed in [51]. Under mild assumptions on the estimation mapping, we show that this approximation is a weakly differentiable function of the parameters and its weak gradient, coined the Stein Unbiased GrAdient estimator of the Risk (SUGAR), provides an asymptotically (with respect to the data dimension) unbiased estimate of the gradient of the risk. Moreover, in the particular case of softthresholding, it is proved to be also a consistent estimator. This gradient estimate can then be used as a basis to perform a quasi-Newton optimization. The computation of the SUGAR relies on the closed-form (weak) differentiation of the non-smooth function. We provide its expression for a large class of iterative methods including proximal splitting ones and apply our strategy to regularizations involving non-smooth convex structured penalties. Illustrations on various image restoration and matrix completion problems are given.

show abstract

“…where d lava (y, X) is the lava estimator on the data (y, X) and d lasso (K 1/2 λ 2 y, K 1/2 λ 2 X)) is the lasso estimator on the data (K 1/2 λ 2 y, K 1/2 λ 2 X) with the penalty level λ 1 . The almost differentiability of the map y → d lasso (K 1/2 λ 2 y, K 1/2 λ 2 X) follows from the almost differentiability of the map u → d lasso (u, K 1/2 λ 2 X), which holds by the results in Dossal et al (2011) and Tibshirani and Taylor (2012).…”

Section: Degrees Of Freedom and Surementioning

confidence: 73%

A lava attack on the recovery of sums of dense and sparse signals

Chernozhukov¹,

Hansen²,

Liao³

2015

View full text Add to dashboard Cite

Abstract. Common high-dimensional methods for prediction rely on having either a sparse signal model, a model in which most parameters are zero and there are a small number of non-zero parameters that are large in magnitude, or a dense signal model, a model with no large parameters and very many small non-zero parameters. We consider a generalization of these two basic models, termed here a "sparse+dense" model, in which the signal is given by the sum of a sparse signal and a dense signal. Such a structure poses problems for traditional sparse estimators, such as the lasso, and for traditional dense estimation methods, such as ridge estimation. We propose a new penalization-based method, called lava, which is computationally efficient. With suitable choices of penalty parameters, the proposed method strictly dominates both lasso and ridge. We derive analytic expressions for the finite-sample risk function of the lava estimator in the Gaussian sequence model. We also provide a deviation bound for the prediction risk in the Gaussian regression model with fixed design. In both cases, we provide Stein's unbiased estimator for lava's prediction risk. A simulation example compares the performance of lava to lasso, ridge, and elastic net in a regression example using feasible, data-dependent penalty parameters and illustrates lava's improved performance relative to these benchmarks.

show abstract

The degrees of freedom of the Lasso for general design matrix

Cited by 24 publications

References 37 publications

Risk estimators for choosing regularization parameters in ill-posed problems - properties and limitations

Risk estimators for choosing regularization parameters in ill-posed problems - properties and limitations

Stein Unbiased GrAdient estimator of the Risk (SUGAR) for Multiple Parameter Selection

A lava attack on the recovery of sums of dense and sparse signals

Contact Info

Product

Resources

About