Minimization of transformed $$L_1$$ L 1 penalty: theory, difference of convex function algorithm, and robust application in compressed sensing

Abstract

We study the minimization problem of a non-convex sparsity promoting penalty function, the transformed l 1 (TL1), and its application in compressed sensing (CS). The TL1 penalty interpolates l 0 and l 1 norms through a nonnegative parameter a ∈ (0, +∞), similar to lp with p ∈ (0, 1], and is known to satisfy unbiasedness, sparsity and Lipschitz continuity properties. We first consider the constrained minimization problem, and discuss the exact recovery of l 0 norm minimal solution based on the null space property (NSP). We then prove the stable recovery of l 0 norm minimal solution if the sensing matrix A satisfies a restricted isometry property (RIP). We formulated a normalized problem to overcome the lack of scaling property of the TL1 penalty function. For a general sensing matrix A, we show that the support set of a local minimizer corresponds to linearly independent columns of A. Next, we present difference of convex algorithms for TL1 (DCATL1) in computing TL1-regularized constrained and unconstrained problems in CS. The DCATL1 algorithm involves outer and inner loops of iterations, one time matrix inversion, repeated shrinkage operations and matrix-vector multiplications. The inner loop concerns an l 1 minimization problem on which we employ the Alternating Direction Method of Multipliers (ADMM). For the unconstrained problem, we prove convergence of DCATL1 to a stationary point satisfying the first order optimality condition. In numerical experiments, we identify the optimal value a = 1, and compare DCATL1 with other CS algorithms on two classes of sensing matrices: Gaussian random matrices and over-sampled discrete cosine transform matrices (DCT). Among existing algorithms, the iterated reweighted least squares method based on l 1/2 norm is the best in sparse recovery for Gaussian matrices, and the DCA algorithm based on l 1 minus l 2 penalty is the best for over-sampled DCT matrices. We find that for both classes of sensing matrices, the performance of DCATL1 algorithm (initiated with l 1 minimization) always ranks near the top (if not the top), and is the most robust choice insensitive to the conditioning of the sensing matrix A. DCATL1 is also competitive in comparison with DCA on other non-convex penalty functions commonly used in statistics with two hyperparameters.

A Scale-Invariant Approach for Sparse Signal Recovery

“…The inequality (1.6) implies that L 1 may not perform well for highly coherent matrices, i.e., µ(A) ∼ 1, as x 0 is then at most one, which seldom occurs simultaneously with Ax * = b. Other than the popular L 1 norm, there are a variety of regularization functionals to promote sparsity, such as L p [9,43,23], L 1 -L 2 [44,26], capped L 1 (CL1) [48,37], and transformed L 1 (TL1) [29,46,47]. Most of these models are nonconvex, leading to difficulties in proving exact recovery guarantees and algorithmic convergence, but they tend to give better empirical results compared to the convex L 1 approach.…”

“…For example, it was reported in [44,26] that L p gives superior results for incoherent matrices (i.e., µ(A) is small), while L 1 -L 2 is the best for the coherent scenario. In addition, TL1 is always the second best no matter whether the matrix is coherent or not [46,47].…”

Transformedℓ1regularization for learning sparse deep neural networks

“…Obviously, when a → 0, i min(|x i |, a)/a → x 0 . Transformed 1 , which is a smooth version of capped 1 , is discussed in the works [39][40][41]. Some other non-convex metrics with concise form are also considered as alternatives to improve 1 , including p with p ∈ (0, 1) [33][34][35], whose formula is…”

“…Although 1 enjoys several good properties, it is sensitive to outliers and may cause serious bias in estimation [26,27]. To overcome this defect, many non-convex surrogates are proposed and analyzed, including smoothly clipped absolute deviation (SCAD) [26], log penalty [28,29], capped 1 [30,31], minimax concave penalty (MCP) [32], p penalty with p ∈ (0, 1) [33][34][35], the difference of 1 and 2 norms [36][37][38] and transformed 1 [39][40][41]. More and more works have shown the good performance of non-convex regularizers in both theoretical analyses and practical applications.…”

A Study on Graph-Structured Recurrent Neural Networks and Sparsification with Application to Epidemic Forecasting

“…Since lim a→0 + ρ a (x i ) = 1 {xi =0} , lim a→+∞ ρ a (x i ) = |x i |, ∀i, the T 1 penalty interpolates 1 and 0 . For its sparsification in compressed sensing and other applications, see [14] and references therein. To sparsify weights in GSRNN training via 1 and T 1 , we add them to the loss function of GSRNN with a multiplicative penalty parameter α > 0, and call stochastic gradient descent optimizer on Tensorflow.…”