Optimal $k$-Thresholding Algorithms for Sparse Optimization Problems

Zhao, Yun-Bin

doi:10.1137/18m1219187

Cited by 37 publications

(124 citation statements)

References 54 publications

Supporting

Mentioning

117

Contrasting

Order By: Relevance

“…Compared with SVRGHT, we can see that the results of our SRGSP in the first few iterations are similar to SVRGHT. However, due to lots of gradient updates followed by a hard thresholding, SRGSP can obtain a better solution, as discussed in [ 36 ]. This further verify the advantage of our SRGSP against other methods.…”

Section: Resultsmentioning

confidence: 99%

“…In other words, we use the stochastic recursive gradient proposed in [ 35 ], which is suitable for solving non-convex problems, to optimize the non-convex sparse representation problem in this paper. In order to keep the gradient information of current iterate as suggested in [ 36 ], we perform lots of gradient descent steps, followed by a hard thresholding operation. We also construct the most relevant support on which minimization will be efficient.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Stochastic Recursive Gradient Support Pursuit and Its Sparse Representation Applications

Shang

Wei

Liu

et al. 2020

Sensors

View full text Add to dashboard Cite

In recent years, a series of matching pursuit and hard thresholding algorithms have been proposed to solve the sparse representation problem with ℓ0-norm constraint. In addition, some stochastic hard thresholding methods were also proposed, such as stochastic gradient hard thresholding (SG-HT) and stochastic variance reduced gradient hard thresholding (SVRGHT). However, each iteration of all the algorithms requires one hard thresholding operation, which leads to a high per-iteration complexity and slow convergence, especially for high-dimensional problems. To address this issue, we propose a new stochastic recursive gradient support pursuit (SRGSP) algorithm, in which only one hard thresholding operation is required in each outer-iteration. Thus, SRGSP has a significantly lower computational complexity than existing methods such as SG-HT and SVRGHT. Moreover, we also provide the convergence analysis of SRGSP, which shows that SRGSP attains a linear convergence rate. Our experimental results on large-scale synthetic and real-world datasets verify that SRGSP outperforms state-of-the-art related methods for tackling various sparse representation problems. Moreover, we conduct many experiments on two real-world sparse representation applications such as image denoising and face recognition, and all the results also validate that our SRGSP algorithm obtains much better performance than other sparse representation learning optimization methods in terms of PSNR and recognition rates.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Stochastic Recursive Gradient Support Pursuit and Its Sparse Representation Applications

Shang

Wei

Liu

et al. 2020

Sensors

View full text Add to dashboard Cite

show abstract

“…Clearly, the convexity of f (λ 6 ) guarantees that ( 37) is a convex optimization. Moreover, (36) and the property…”

Section: Relaxation Modelsmentioning

confidence: 99%

“…Due to the constraints (36), the optimal value of the problem ( 37) is finite if it is feasible. By replacing ζ by R n + in (37) , we also obtain a new relaxation of ( 22):…”

Section: One-step Dual-density-based Algorithmmentioning

confidence: 99%

“…The reweighted 1minimization may perform better than 1 -minimization on sparse signal recovery when the initial point is suitably chosen (see, e.g., [8,9,15,22,35,38]). Although this paper focuses on the study of reweighted algorithms, it is worth mentioning that there exist other types of algorithms for 0 -minimization problems, which have also been widely studied in the CS literature, such as orthogonal matching pursuits [14,25,30], compressed sampling matching pursuits [16,27], subspace pursuits [10,16], thresholding algorithms [3,11,14,16,26], and the newly developed optimal k-thresholding algorithms [36].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Dual-density-based reweighted $$\ell _{1}$$-algorithms for a class of $$\ell _{0}$$-minimization problems

Zhao

2021

J Glob Optim

Self Cite

View full text Add to dashboard Cite

The optimization problem with sparsity arises in many areas of science and engineering such as compressed sensing, image processing, statistical learning and data sparse approximation. In this paper, we study the dual-density-based reweighted $$\ell _{1}$$ ℓ 1 -algorithms for a class of $$\ell _{0}$$ ℓ 0 -minimization models which can be used to model a wide range of practical problems. This class of algorithms is based on certain convex relaxations of the reformulation of the underlying $$\ell _{0}$$ ℓ 0 -minimization model. Such a reformulation is a special bilevel optimization problem which, in theory, is equivalent to the underlying $$\ell _{0}$$ ℓ 0 -minimization problem under the assumption of strict complementarity. Some basic properties of these algorithms are discussed, and numerical experiments have been carried out to demonstrate the efficiency of the proposed algorithms. Comparison of numerical performances of the proposed methods and the classic reweighted $$\ell _1$$ ℓ 1 -algorithms has also been made in this paper.

show abstract

On convergence of iterative thresholding algorithms to approximate sparse solution for composite nonconvex optimization

Hu,

Yang

2024

Math. Program.

View full text Add to dashboard Cite

This paper aims to find an approximate true sparse solution of an underdetermined linear system. For this purpose, we propose two types of iterative thresholding algorithms with the continuation technique and the truncation technique respectively. We introduce a notion of limited shrinkage thresholding operator and apply it, together with the restricted isometry property, to show that the proposed algorithms converge to an approximate true sparse solution within a tolerance relevant to the noise level and the limited shrinkage magnitude. Applying the obtained results to nonconvex regularization problems with SCAD, MCP and $$\ell _p$$ ℓ p penalty ($$0\le p \le 1$$ 0 ≤ p ≤ 1 ) and utilizing the recovery bound theory, we establish the convergence of their proximal gradient algorithms to an approximate global solution of nonconvex regularization problems. The established results include the existing convergence theory for $$\ell _1$$ ℓ 1 or $$\ell _0$$ ℓ 0 regularization problems for finding a true sparse solution as special cases. Preliminary numerical results show that our proposed algorithms can find approximate true sparse solutions that are much better than stationary solutions that are found by using the standard proximal gradient algorithm.

show abstract

Optimal $k$-Thresholding Algorithms for Sparse Optimization Problems

Cited by 37 publications

References 54 publications

Stochastic Recursive Gradient Support Pursuit and Its Sparse Representation Applications

Stochastic Recursive Gradient Support Pursuit and Its Sparse Representation Applications

Dual-density-based reweighted $$\ell _{1}$$-algorithms for a class of $$\ell _{0}$$-minimization problems

On convergence of iterative thresholding algorithms to approximate sparse solution for composite nonconvex optimization

Contact Info

Product

Resources

About