A multidimensional shrinkage-thresholding operator

Abstract

The scalar shrinkage-thresholding operator (SSTO) is a key ingredient of many modern statistical signal processing algorithms including: sparse inverse problem solutions, wavelet denoising, and JPEG2000 image compression. In these applications, it is customary to select the threshold of the operator by solving a scalar sparsity penalized quadratic optimization. In this work, we present a natural multidimensional extension of the scalar shrinkage thresholding operator. Similarly to the scalar case, the threshold is determined by the minimization of a convex quadratic form plus an euclidean penalty, however, here the optimization is performed over a domain of dimension N ≥ 1. The solution to this convex optimization problem is called the multidimensional shrinkage threshold operator (MSTO). The MSTO reduces to the standard SSTO in the special case of N = 1. In the general case of N > 1 the optimal MSTO threshold can be found by a simple convex line search. We present three illustrative applications of the MSTO in the context of non-linear regression: l 2 -penalized linear regression, Group LASSO linear regression and Group LASSO logistic regression. In this paper, we introduce a multidimensional generalization of the scalar shrinkage thresholding operator. We define this operator as the minimization of a convex quadratic form plus an Euclidean norm penalty. We analyze this nondifferentiable optimization problem and discuss its properties. In particular, in analogy to the scalar shrinkage operator, we show that this generalization yields a multidimensional Shrinkage Thresholding Operator (MSTO) which takes a vector as an input and shrinks it or thresholds it depending on its Euclidean norm. For this purpose, we reformulate the problem as a constrained quadratic problem with a conic constraint. This principle leads to a theoretical result that transforms this multidimensional optimization problem into a simple line search which can be efficiently implemented. We show by simulations that evaluating the MSTO using line search is competitive with state-of-the-art convex solvers.In the second part of the paper, we discuss applications of the MSTO to statistical regression. First, we consider the Euclidean-norm penalized least squares and discuss its relation to ridge regression and robust regression [5]. Next, we address group LASSO linear regression [6]. In the special case of a block-orthogonal design matrix, we show that the problem can be reduced to evaluating the MSTO for each block. For other Group LASSO problems, we propose two iterative applications of the MSTO. In the first approach, we use Block Coordinate Descent to solve the linear regression problem with an arbitrary design matrix. The second approach tackles more complicated cost functions such as the logistic regression objective. Due to its similarity to the wellknown class of Iterative Thresholding Algorithms [1], we name the latter Iterative Group Shrinkage-Thresholding (IGST). In both cases, the MSTO enables one to solve large scale Group LASS...

Group-Lasso on Splines for Spectrum Cartography

“…As such, result (21) can be viewed as a particular case of the operators in [22] and [28]. However it is worth to prove Lemma 2 directly, since in this case the special form of (20) renders the proof neat in its simplicity.…”

“…Different from [29], GLasso can handle a general (not orthonormal) regression matrix X. Compared to the block-coordinate algorithm of [22], GLasso does not require an inner Newton-Raphson recursion per iteration. If in addition N b = p, then GLasso yields the Lasso estimator.…”

“…Group-Lasso estimators are known to set groups of weak coefficients to zero (here the N b groups associated with coefficients per g ν ), and outperform the sparsity-agnostic LS estimator by capitalizing on the sparsity present [29], [22]. An iterative group-Lasso algorithm is developed that yields closed-form estimates per iteration.…”

“…(24)- (27): Recall the augmented Lagrangian function in (23), and let ζ := {ζ r } r∈R for notational brevity. When used to solve (22), the three steps in the AD-MoM are given by:…”

Group lassoing change-points in piecewise-constant AR processes

“…n can be obtained via interior point methods or by (numerically) solving the scalar equation in (21), which admits fast solvers via, e.g., Newton-Raphson iterations, as in [25].…”