Fast gradient methods with alignment for symmetric linear systems without using Cauchy step

Zou, Qinmeng; Magoulès, Frédéric

doi:10.1016/j.cam.2020.113033

Cited by 7 publications

(6 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We remark that there also exist a variety of modifications and extensions of the SD method and the BB method in the literature, including gradient methods with retards [19], gradient methods with alignment [15,16], alternate step gradient methods [6,10,14], cyclic SD method [9], cyclic BB method [11], adaptive BB method [18], limited memory gradient method [5] etc; see [3,12,20,23,31,32] and references therein for other researches on the choices of the stepsize k in the gradient method (2).…”

Section: Introductionmentioning

confidence: 99%

On R-linear convergence analysis for a class of gradient methods

Huang

2021

Comput Optim Appl

View full text Add to dashboard Cite

Gradient method is a simple optimization approach using minus gradient of the objective function as a search direction. Its efficiency highly relies on the choices of the stepsize. In this paper, the convergence behavior of a class of gradient methods, where the stepsize has an important property introduced in (Dai in Optimization 52: 2003), is analyzed. Our analysis is focused on minimization on strictly convex quadratic functions. We establish the R-linear convergence and derive an estimate for the R-factor. Specifically, if the stepsize can be expressed as a collection of Rayleigh quotient of the inverse Hessian matrix, we are able to show that these methods converge R-linearly and their R-factors are bounded above by 1 − 1 , where is the associated condition number. Preliminary numerical results demonstrate the tightness of our estimate of the R-factor.

show abstract

Section: Introductionmentioning

confidence: 99%

On R-linear convergence analysis for a class of gradient methods

Huang

2021

Comput Optim Appl

View full text Add to dashboard Cite

show abstract

“…It is proved that the proposed method is R-linearly convergent with the rate of 1 − 1/κ, where κ = λ n /λ 1 is the condition number of A. Our numerical comparisons with the BB1 [2], DY [14], SL (Alg.1 in [35]), ABBmin2 [21], SDC [16], and MGC [39] methods for solving unconstrained random and non-random quadratic optimization demonstrate that the proposed method is very efficient. Further, numerical experiments on quadratic problems whose Hessians are chosen from the SuiteSparse Matrix Collection [15] suggest that the proposed method is very competitive with the above methods.…”

Section: Introductionmentioning

confidence: 81%

“…Based on the above analysis, we can develop gradient method using αk and α BB1 k in an adaptive way. Notice that reusing the retard short stepsize for some iterations could reduce the computational cost and yield better performance, see [16,27,35,37,39]. So, we suggest to combine the adaptive and cyclic schemes with α BB1 k and αk−1 .…”

Section: ⊓ ⊔mentioning

confidence: 99%

See 1 more Smart Citation

A gradient method exploiting the two dimensional quadratic termination property

Li¹,

Huang²

2022

Preprint

View full text Add to dashboard Cite

The quadratic termination property is important to the efficiency of gradient methods. We consider equipping a family of gradient methods, where the stepsize is given by the ratio of two norms, with two dimensional quadratic termination. Such a desired property is achieved by cooperating with a new stepsize which is derived by maximizing the stepsize of the considered family in the next iteration. It is proved that each method in the family will asymptotically alternate in a two dimensional subspace spanned by the eigenvectors corresponding to the largest and smallest eigenvalues. Based on this asymptotic behavior, we show that the new stepsize converges to the reciprocal of the largest eigenvalue of the Hessian. Furthermore, by adaptively taking the long Barzilai-Borwein stepsize and reusing the new stepsize with retard, we propose an efficient gradient method for unconstrained quadratic optimization. We prove that the new method is R-linearly convergent with a rate of 1 − 1/κ, where κ is the condition number of Hessian. Numerical experiments show the efficiency of our proposed method.

show abstract

“…Successful applications of the BB gradient method and its variants have been found in sparse reconstruction [39,41], nonnegative matrix factorization [32], optimization on manifolds [22,33,34,40], machine learning [11,38,42], etc. Since optimization problems arising in different areas are often in large scale, BB-like methods have attracted increasingly attention in recent years due to their simplicity, numerical efficiency and low per-iteration cost, see [9,12,13,17,19,21,23,36,44,46,47] and references therein.…”

Section: Introductionmentioning

confidence: 99%

Equipping Barzilai-Borwein method with two dimensional quadratic termination property

Huang

Dai

2020

Preprint

View full text Add to dashboard Cite

The Barzilai-Borwein (BB) gradient method, computing the step size by imposing certain quasi-Newton property, is more efficient than the classic steepest descent (SD) method though it generally produces nonmonotone sequences. It is observed that the SD method adopts the Yuan stepsize is very competitive with the BB gradient method since the Yuan stepsize yields finite termination for two-dimensional strictly convex quadratic functions. In this paper, we investigate to accelerate the BB gradient method by imposing the above two-dimensional quadratic termination property. We introduce a novel equation and show that in the two-dimensional quadratic case it is equivalent to impose the gradient aligns with some eigendirection of the Hessian at next iteration, which will yield finite termination. Moreover, the Yuan stepsize can be recovered as a special solution of the equation. Based on the proposed equation, we derive a new stepsize for the BB gradient method. In addition to the aforementioned two-dimensional quadratic termination property, a remarkable feature of the new stepsize is that its computation only depends on the long and short BB stepsizes in two consecutive iterations, without the need for exact line search and Hessian. It thus has a great advantage of being easily extended

show abstract

Fast gradient methods with alignment for symmetric linear systems without using Cauchy step

Cited by 7 publications

References 33 publications

On R-linear convergence analysis for a class of gradient methods

On R-linear convergence analysis for a class of gradient methods

A gradient method exploiting the two dimensional quadratic termination property

Equipping Barzilai-Borwein method with two dimensional quadratic termination property

Contact Info

Product

Resources

About