Multidimensional Shrinkage-Thresholding Operator and Group LASSO Penalties

Abstract

The scalar shrinkage-thresholding operator (SSTO) is a key ingredient of many modern statistical signal processing algorithms including: sparse inverse problem solutions, wavelet denoising, and JPEG2000 image compression. In these applications, it is customary to select the threshold of the operator by solving a scalar sparsity penalized quadratic optimization. In this work, we present a natural multidimensional extension of the scalar shrinkage thresholding operator. Similarly to the scalar case, the threshold is determined by the minimization of a convex quadratic form plus an euclidean penalty, however, here the optimization is performed over a domain of dimension N ≥ 1. The solution to this convex optimization problem is called the multidimensional shrinkage threshold operator (MSTO). The MSTO reduces to the standard SSTO in the special case of N = 1. In the general case of N > 1 the optimal MSTO threshold can be found by a simple convex line search. We present three illustrative applications of the MSTO in the context of non-linear regression: l 2 -penalized linear regression, Group LASSO linear regression and Group LASSO logistic regression. In this paper, we introduce a multidimensional generalization of the scalar shrinkage thresholding operator. We define this operator as the minimization of a convex quadratic form plus an Euclidean norm penalty. We analyze this nondifferentiable optimization problem and discuss its properties. In particular, in analogy to the scalar shrinkage operator, we show that this generalization yields a multidimensional Shrinkage Thresholding Operator (MSTO) which takes a vector as an input and shrinks it or thresholds it depending on its Euclidean norm. For this purpose, we reformulate the problem as a constrained quadratic problem with a conic constraint. This principle leads to a theoretical result that transforms this multidimensional optimization problem into a simple line search which can be efficiently implemented. We show by simulations that evaluating the MSTO using line search is competitive with state-of-the-art convex solvers.In the second part of the paper, we discuss applications of the MSTO to statistical regression. First, we consider the Euclidean-norm penalized least squares and discuss its relation to ridge regression and robust regression [5]. Next, we address group LASSO linear regression [6]. In the special case of a block-orthogonal design matrix, we show that the problem can be reduced to evaluating the MSTO for each block. For other Group LASSO problems, we propose two iterative applications of the MSTO. In the first approach, we use Block Coordinate Descent to solve the linear regression problem with an arbitrary design matrix. The second approach tackles more complicated cost functions such as the logistic regression objective. Due to its similarity to the wellknown class of Iterative Thresholding Algorithms [1], we name the latter Iterative Group Shrinkage-Thresholding (IGST). In both cases, the MSTO enables one to solve large scale Group LASS...

Efficient Methods to Compute Optimal Tree Approximations of Directed Information Graphs

“…Consider the joint distribution of random processes , each of length . We will consider approximations of the form (8) where selects the parent. Let denote the set of all such approximations.…”

“…"Group Lasso" is a method to infer the causal relationships between multivariate auto-regressive models [6]. Bolstad [8]. Tan and Willsky analyzed sample complexity for identifying the topology of a tree structured network of LTI systems [9].…”

Decentralized Optimal Dispatch of Photovoltaic Inverters in Residential Distribution Systems

“…Unique to the OID formulation are the binary optimization variables {z h }; finding the optimal (sub)set of inverters to dispatch involves the solution of combinatorially many subproblems. Nevertheless, a computationallyaffordable convex reformulation was developed in [11], by leveraging sparsity-promoting regularization [26] and semidefinite relaxation (SDR) techniques [12], [23], [27] as briefly described next.…”

“…Specifically, the number of inverters operating under OID decreases as λ is increased [26]. Key to developing a relaxation of the OID task is to express powers and voltage magnitudes as linear functions of the outerproduct Hermitian matrix V := vv H , and to reformulate the OID problem with cost and constraints that are linear in V, as well as the constraints V 0 and rank(V) = 1 [12], [23], [27].…”

“…As for the inverter setpoints {(P av h − P c,h , Q c,h )}, those obtained from (4) may be slightly sub-optimal compared to the setpoints that would have been obtained by solving the optimization problem (2). This is mainly due to the socalled "shrinkage effect" introduced by the regularizer (3) [26]. Unfortunately, a numerical assessment of the optimality gap is impractical, since finding the globally optimal solution of problem (2) under all setups is computationally infeasible.…”