On Convergence of Projected Gradient Descent for Minimizing a Large-Scale Quadratic Over the Unit Sphere

Vu, Trung; Raich, Raviv; Fu, Xiao

doi:10.1109/mlsp.2019.8918830

Cited by 9 publications

(23 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Remark 6. The local linear rate in (50) matches the rate provided by Theorem 1 in [35]. Compared to the setting in [35], here we consider a special case of the quadratic that is convex (and hence, λ d ≥ 0).…”

Section: Least Squares With the Unit Norm Constraintmentioning

confidence: 76%

“…which means Ax * − b is in the left null space of AV ⊥ C . 3 Step 3: Evaluating the projection in (33) at z * η = x * − ηA ⊤ (Ax * − b) and using the stationarity condition (35) to eliminate the term…”

Section: A Linearly Constrained Least Squaresmentioning

confidence: 99%

“…where γ = (x * ) ⊤ A ⊤ (Ax * − b) is the Lagrange multiplier at x * (see Lemma 1 in [35]). Thus, we obtain the condition for x * ∈ C to be a Lipschitz stationary point of ( 46) is x * and A ⊤ (Ax * − b) are collinear.…”

Section: Least Squares With the Unit Norm Constraintmentioning

confidence: 99%

“…Compared to the setting in [35], here we consider a special case of the quadratic that is convex (and hence, λ d ≥ 0). By minimizing the rate in (50) over η, we also obtain the same optimal rate of linear convergence given by Lemma 5 in [35]:…”

Section: Least Squares With the Unit Norm Constraintmentioning

confidence: 99%

See 3 more Smart Citations

On Asymptotic Linear Convergence of Projected Gradient Descent for Constrained Least Squares

Vu,

Raich

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Many recent problems in signal processing and machine learning such as compressed sensing, image restoration, matrix/tensor recovery, and non-negative matrix factorization can be cast as constrained optimization. Projected gradient descent is a simple yet efficient method for solving such constrained optimization problems. Local convergence analysis furthers our understanding of its asymptotic behavior near the solution, offering sharper bounds on the convergence rate compared to global convergence analysis. However, local guarantees often appear scattered in problem-specific areas of machine learning and signal processing. This manuscript presents a unified framework for the local convergence analysis of projected gradient descent in the context of constrained least squares. The proposed analysis offers insights into pivotal local convergence properties such as the condition of linear convergence, the region of convergence, the exact asymptotic rate of convergence, and the bound on the number of iterations needed to reach a certain level of accuracy. To demonstrate the applicability of the proposed approach, we present a recipe for the convergence analysis of PGD and demonstrate it via a beginning-to-end application of the recipe on four fundamental problems, namely, linearly constrained least squares, sparse recovery, least squares with the unit norm constraint, and matrix completion.

show abstract

Section: Least Squares With the Unit Norm Constraintmentioning

confidence: 76%

Section: A Linearly Constrained Least Squaresmentioning

confidence: 99%

Section: Least Squares With the Unit Norm Constraintmentioning

confidence: 99%

Section: Least Squares With the Unit Norm Constraintmentioning

confidence: 99%

See 2 more Smart Citations

On Asymptotic Linear Convergence of Projected Gradient Descent for Constrained Least Squares

Vu,

Raich

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…where 0 > 0, 0 < < 1, and ≥ 0 are real scalars. In optimization, equations of form (1) emanates from the convergence analysis of iterative first-order methods [6][7][8][9][10], in which the sequence { } ∞ =0 represents the error through iterations (e.g., the distance from the current update to the optimum). Naturally, the convergence of { } ∞ =0 to zero provides guarantees on the performance of these optimization algorithms.…”

Section: Introductionmentioning

confidence: 99%

A Closed-Form Bound on the Asymptotic Linear Convergence of Iterative Methods via Fixed Point Analysis

Vu¹,

Raich²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

We present a tight closed-form bound on the convergence of the quadratic first-order difference equation with positive coefficients. Our result offers insight into the convergence behavior of the dynamic system in its convergent regime: while the quadratic term is negligible in the asymptotic regime, it leads to a finite increase in the number of iterations required for a given accuracy relative to the corresponding linear difference equation.

show abstract

A closed-form bound on the asymptotic linear convergence of iterative methods via fixed point analysis

Raich

2022

Optim Lett

View full text Add to dashboard Cite

On Convergence of Projected Gradient Descent for Minimizing a Large-Scale Quadratic Over the Unit Sphere

Cited by 9 publications

References 9 publications

On Asymptotic Linear Convergence of Projected Gradient Descent for Constrained Least Squares

On Asymptotic Linear Convergence of Projected Gradient Descent for Constrained Least Squares

A Closed-Form Bound on the Asymptotic Linear Convergence of Iterative Methods via Fixed Point Analysis

A closed-form bound on the asymptotic linear convergence of iterative methods via fixed point analysis

Contact Info

Product

Resources

About