Global convergence result for conjugate gradient methods

Hu, Yaping; Storey, C.

doi:10.1007/bf00939927

Cited by 143 publications

(64 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [31], Hu and Storey proposed a hybrid conjugate gradient method (HY-HuSt) by combining the good numerical performance of Polak-Ribière (PR) method and the nice global con-vergence properties of Fletcher-Reeves (FR) method. Subsequently, Gilbert and Nocedal [32] suggested a modified version of Hu-Storey method (HY-GN).…”

Section: B Conjugate Gradient Methodsmentioning

confidence: 99%

Efficient Training Algorithms for a Class of Shunting Inhibitory Convolutional Neural Networks

Tivive

Bouzerdoum

2005

IEEE Trans. Neural Netw.

View full text Add to dashboard Cite

This article presents some efficient training algorithms, based on first-order, second-order, and conjugate gradient optimization methods, for a class of convolutional neural networks (CoNNs), known as shunting inhibitory convolution neural networks. Furthermore, a new hybrid method is proposed, which is derived from the principles of Quickprop, Rprop, SuperSAB, and least squares (LS). Experimental results show that the new hybrid method can perform as well as the Levenberg-Marquardt (LM) algorithm, but at a much lower computational cost and less memory storage. For comparison sake, the visual pattern recognition task of face/nonface discrimination is chosen as a classification problem to evaluate the performance of the training algorithms. Sixteen training algorithms are implemented for the three different variants of the proposed CoNN architecture: binary-, Toeplitz-and fully connected architectures. All implemented algorithms can train the three network architectures successfully, but their convergence speed vary markedly. In particular, the combination of LS with the new hybrid method and LS with the LM method achieve the best convergence rates in terms of number of training epochs. In addition, the classification accuracies of all three architectures are assessed using ten-fold cross validation. The results show that the binary-and Toeplitzconnected architectures outperform slightly the fully connected architecture: the lowest error rates across all training algorithms are 1.95% for Toeplitz-connected, 2.10% for the binary-connected, and 2.20% for the fully connected network. In general, the modified Broyden-Fletcher-Goldfarb-Shanno (BFGS) methods, the three variants of LM algorithm, and the new hybrid/LS method perform consistently well, achieving error rates of less than 3% averaged across all three architectures.

show abstract

Section: B Conjugate Gradient Methodsmentioning

confidence: 99%

Efficient Training Algorithms for a Class of Shunting Inhibitory Convolutional Neural Networks

Tivive

Bouzerdoum

2005

IEEE Trans. Neural Netw.

View full text Add to dashboard Cite

show abstract

“…In addition, the hybrid methods in [6,16,19] can also be regarded as special cases of the three-parameter family. For example, to combine the nice global convergence properties of the FR method and the good numerical performances of the PRP method, Hu and Storey [19] proposed a hybrid method, where…”

Section: A Three-parameter Family Of Conjugate Gradient Methodsmentioning

confidence: 99%

“…For example, to combine the nice global convergence properties of the FR method and the good numerical performances of the PRP method, Hu and Storey [19] proposed a hybrid method, where…”

Section: A Three-parameter Family Of Conjugate Gradient Methodsmentioning

confidence: 99%

A three-parameter family of nonlinear conjugate gradient methods

Dai¹,

Yuan²

2000

Math. Comp.

View full text Add to dashboard Cite

Abstract. In this paper, we propose a three-parameter family of conjugate gradient methods for unconstrained optimization. The three-parameter family of methods not only includes the already existing six practical nonlinear conjugate gradient methods, but subsumes some other families of nonlinear conjugate gradient methods as its subfamilies. With Powell's restart criterion, the three-parameter family of methods with the strong Wolfe line search is shown to ensure the descent property of each search direction. Some general convergence results are also established for the three-parameter family of methods. This paper can also be regarded as a brief review on nonlinear conjugate gradient methods.

show abstract

“…Hybrid conjugate gradient algorithms using projections: hybrid Dai-Yuan [23], Gilbert and Nocedal [28], Hu and Storey [34], Touati-Ahmed and Storey [50], hybrid Liu and Storey [36], and hybrid conjugate gradient algorithms using the concept of convex combination of classical schemes: convex combination of Hestenes-Stiefel and Dai-Yuan with Newton direction [3,4,8], convex combination of Polak-Ribière-Polyak and DaiYuan with conjugacy condition [7]. Scaled BFGS preconditioned conjugate gradient algorithms by Shanno [47,48], Birgin and Martínez [18] and Andrei [2,9].…”

Section: Classical Conjugate Gradient Algorithmsmentioning

confidence: 99%

A Numerical Study on Efficiency and Robustness of Some Conjugate Gradient Algorithms for Large-scale Unconstrained Optimization

Andrei¹

2013

SIC

View full text Add to dashboard Cite

Abstract:A numerical evaluation and comparisons using performance profiles of some representative conjugate gradient algorithms for solving a large variety of large-scale unconstrained optimization problems are carried on. In this intensive numerical study we selected eight known conjugate gradient algorithms: Hestenes and Stiefel (HS), Polak-Ribière-Polyak (PRP), CONMIN, ASCALCG, CG-DESCENT, AHYBRIDM, THREECG and DESCON. These algorithms are different in many respects. However, they have a lot of concepts in common, which give the numerical comparisons sense and confident expectations. The initial search direction in all algorithms is the negative gradient computed in the initial point and the step length is computed by the Wolfe line search conditions. Excepting CONMIN and CG-DESCENT, all the algorithms from this numerical study implement an acceleration scheme which modifies the step length in a multiplicative manner to improve the reduction of the functions values along the iterations. The numerical study is based on a set of 800 artificially large-scale unconstrained optimization test functions of different complexity and with different structures of their Hessian matrix. A detailed numerical evaluation based on performance profiles is applied to the comparisons of these algorithms showing that all of them are able to solve difficult large-scale unconstrained optimization problems. However, comparisons using only artificially test problems are weak and dependent on arbitrary choices concerning the stopping criteria of the algorithms and on decision of whether an algorithm found a solution or not. To get definitive conclusions using this sort of comparisons based only on artificially test problems is an illusion. However, using some real unconstrained optimization applications we can get a more confident conclusion about the efficiency and robustness of optimization algorithms considered in this numerical study.

show abstract

Global convergence result for conjugate gradient methods

Cited by 143 publications

References 2 publications

Efficient Training Algorithms for a Class of Shunting Inhibitory Convolutional Neural Networks

Efficient Training Algorithms for a Class of Shunting Inhibitory Convolutional Neural Networks

A three-parameter family of nonlinear conjugate gradient methods

A Numerical Study on Efficiency and Robustness of Some Conjugate Gradient Algorithms for Large-scale Unconstrained Optimization

Contact Info

Product

Resources

About