Augmented $\ell_1$ and Nuclear-Norm Models with a Globally Linearly Convergent Algorithm

Abstract

This paper studies the long-existing idea of adding a nice smooth function to "smooth" a nondifferentiable objective function in the context of sparse optimization, in particular, the minimization of x 1 + 1 2α x 2 2 , where x is a vector, as well as the minimization of X * + 1 2α X 2 F , where X is a matrix and X * and X F are the nuclear and Frobenius norms of X, respectively. We show that they let sparse vectors and low-rank matrices be efficiently recovered. In particular, they enjoy exact and stable recovery guarantees similar to those known for the minimization of x 1 and X * under the conditions on the sensing operator such as its null-space property, restricted isometry property (RIP), spherical section property, or "RIPless" property. To recover a (nearly) sparse vector x 0 , minimizing x 1 + 1 2α x 2 2 returns (nearly) the same solution as minimizing x 1 whenever α ≥ 10 x 0 ∞. The same relation also holds between minimizing X * + 1 2α X 2 F and minimizing X * for recovering a (nearly) low-rank matrix X 0 if α ≥ 10 X 0 2. Furthermore, we show that the linearized Bregman algorithm, as well as its two fast variants, for minimizing x 1 + 1 2α x 2 2 subject to Ax = b enjoys global linear convergence as long as a nonzero solution exists, and we give an explicit rate of convergence. The convergence property does not require a sparse solution or any properties on A. To the best of our knowledge, this is the best known global convergence result for first-order sparse optimization algorithms. Introduction.Sparse vector recovery and low-rank matrix recovery problems have drawn much attention from researchers in different fields over the past several years. They have wide applications in compressive sensing, signal/image processing, machine learning, etc. The fundamental problem of sparse vector recovery is to find the vector with (nearly) fewest nonzero entries from an underdetermined linear system Ax = b, and that of low-rank matrix recovery is to find a matrix of (nearly) lowest rank from an underdetermined A(X) = b, where A is a linear operator.To recover a sparse vector x 0 , a well-known model is the basis pursuit (BP) problem [12]:

Designing Statistical Estimators That Balance Sample Size, Risk, and Computational Cost

“…For the constant smoothing case, we choose the smoothing parameter µ = 0.1 based on the recommendation in [27] for the noiseless case. It is common, however, to see much smaller choices of µ; see [28], [29].…”

“…Again, current practice dictates using a smoothing parameter that has no dependence on the sample size m; see [31], for example. In our tests, we choose the baseline smoothing parameter µ = 0.1 recommended by [27]. As before, we compare the constant smoothing, constant risk, and balanced (α = 0.9) schemes.…”

Fast singular value thresholding without singular value decomposition

“…It has been shown in [47,76] that the SVT algorithm with a finite τ can get the perfect matrix completion as (2) does. The SVT algorithm is reformulated as Uzawa's algorithm [4] or linearized Bregman iteration [9,10,58,73].…”

Iterative Methods Based on Soft Thresholding of Hierarchical Tensors

“…In such a setting, dual gradient descent methods [40] can offer advantageous convergence behavior, but these exploit that iterates for the dual variable are in a relatively small linear space for completion problems and are thus not appropriate in our situation. In tensor completion, as a replacement of the nuclear norm in the matrix case, the sum of nuclear norms of matricizations has been proposed [32,41]; however, with this approach, one does not recover the particular properties of the matrix case.…”