Accelerating Iterative Hard Thresholding for Low-rank Matrix Completion via Adaptive Restart

Vu, Trung; Raich, Raviv

doi:10.1109/icassp.2019.8683082

Cited by 12 publications

(13 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Assume the same setting as in Theorem 3. As m → ∞, the linear rate ρ defined in (8) converges almost surely to p ∞ defined in (16).…”

Section: Proposed Estimation Of the Linear Rate ρmentioning

confidence: 99%

“…On the one hand, interior-point methods for solving the nuclear norm minimization problem are computationally expensive and even infeasible for large matrices. On the other hand, proximal-type algorithms suffer from slow convergence due to the conservative nature of the soft-thresholding operator [15], [16].…”

Section: Introductionmentioning

confidence: 99%

“…When the solution is low-rank, hard-thresholding algorithms is more efficient than their soft-thresholding counterparts in both computational complexity per iteration and convergence speed. Variants of plain IHT with faster convergence have also been developed, including normalized IHT [19], conjugate gradient IHT [20], Nesterov's accelerated gradient IHT [16], Heavy-Ball IHT [21], just to name a few. The drawback of IHT methods, however, is the lack of mathematical guarantees on their convergence behavior.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Local Convergence of the Heavy Ball Method in Iterative Hard Thresholding for Low-rank Matrix Completion

Raich

2019

ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

Iterative hard thresholding (IHT) has gained in popularity over the past decades in large-scale optimization. However, convergence properties of this method have only been explored recently in non-convex settings. In matrix completion, existing works often focus on the guarantee of global convergence of IHT via standard assumptions such as incoherence property and uniform sampling. While such analysis provides a global upper bound on the linear convergence rate, it does not describe the actual performance of IHT in practice. In this paper, we provide a novel insight into the local convergence of a specific variant of IHT for matrix completion. We uncover the exact linear rate of IHT in a closed-form expression and identify the region of convergence in which the algorithm is guaranteed to converge. Furthermore, we utilize random matrix theory to study the linear rate of convergence of IHTSVD for large-scale matrix completion. We find that asymptotically, the rate can be expressed in closed form in terms of the relative rank and the sampling rate. Finally, we present various numerical results to verify the aforementioned theoretical analysis.

show abstract

“…Assume the same setting as in Theorem 3. As m → ∞, the linear rate ρ defined in (8) converges almost surely to p ∞ defined in (16).…”

Section: Proposed Estimation Of the Linear Rate ρmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Local Convergence of the Heavy Ball Method in Iterative Hard Thresholding for Low-rank Matrix Completion

Raich

2019

ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

show abstract

“…When the size of the matrix grows rapidly, storing and optimizing over a matrix variable become computationally expensive and even infeasible. In addition, it is evident this approach suffers from slow convergence [9,10]. In the second approach, the original rank-constrained optimization is studied.…”

Section: Introductionmentioning

confidence: 99%

“…Thus, basic optimization algorithms such as gradient descent [12,14,15] and alternating minimization [16][17][18][19] can provably solve matrix completion under a specific sampling regime. Alternatively, the original rankconstrained optimization problem can be solved without the aforementioned reparameterization via the truncated singular value decomposition [10,[20][21][22][23][24][25].…”

Section: Introductionmentioning

confidence: 99%

Exact Linear Convergence Rate Analysis for Low-Rank Symmetric Matrix Completion via Gradient Descent

Raich

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

Factorization-based gradient descent is a scalable and efficient algorithm for solving low-rank matrix completion. Recent progress in structured non-convex optimization has offered global convergence guarantees for gradient descent under certain statistical assumptions on the low-rank matrix and the sampling set. However, while the theory suggests gradient descent enjoys fast linear convergence to a global solution of the problem, the universal nature of the bounding technique prevents it from obtaining an accurate estimate of the rate of convergence. This paper performs a local analysis of the exact linear convergence rate of gradient descent for factorization-based symmetric matrix completion. Without any additional assumptions on the underlying model, we identify the deterministic condition for local convergence guarantee for gradient descent, which depends only on the solution matrix and the sampling set. More crucially, our analysis provides a closed-form expression of the asymptotic rate of convergence that matches exactly with the linear convergence observed in practice. To the best of our knowledge, our result is the first one that offers the exact linear convergence rate of gradient descent for matrix factorization in Euclidean space for matrix completion.

show abstract

Convergence of the Mini-Batch SIHT Algorithm

Damadi,

Shen

2024

Lecture Notes in Networks and Systems

View full text Add to dashboard Cite

Accelerating Iterative Hard Thresholding for Low-rank Matrix Completion via Adaptive Restart

Cited by 12 publications

References 15 publications

Local Convergence of the Heavy Ball Method in Iterative Hard Thresholding for Low-rank Matrix Completion

Local Convergence of the Heavy Ball Method in Iterative Hard Thresholding for Low-rank Matrix Completion

Exact Linear Convergence Rate Analysis for Low-Rank Symmetric Matrix Completion via Gradient Descent

Convergence of the Mini-Batch SIHT Algorithm

Contact Info

Product

Resources

About