Quasi-Newton Algorithms with Updates from the Preconvex Part of Broyden's Family

Zhang, Yin; Tewarson, R. P.

doi:10.1093/imanum/8.4.487

Cited by 46 publications

(39 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Zhang and Tewarson [31] usê s = 1 and comment that their negative Broyden algorithm improves iteration counts but that "less or no savings are achieved on the number of function evaluations" because initial steps are often too long to provide a sufficient decrease in the function value. SQN corrects this problem by effectively estimating the optimal step size for the given search direction.…”

Section: M-mentioning

confidence: 99%

“…SQN compares favorably to other studies that have used negative Broyden parameters. Zhang and Tewarson [31] report 21% and 13% fewer iterations for their SDQN method relative to BFGS on problems of small and "increasing" dimension, respectively. However, their improvements were smaller using the EFE metric that incorporates the number of function evaluations.…”

Section: M-mentioning

confidence: 99%

“…Recently, however, various choices of negative Broyden parameters (φ k < 0 corresponding to λ k < 1) have been studied. See, for example, [31], [8], [23], [17], and [26]. These authors report that negative Broyden parameters can reduce iteration counts, although in some cases this comes at the cost of increased numbers of function evaluations.…”

Section: Historical Developments Crockett and Chernoffmentioning

confidence: 99%

“…Robust improvement over BFGS has been elusive. Indeed, Zhang and Tewarson [31] concluded that such investigations have not shaken the position of BFGS as the most popular front-runner.…”

Section: Historical Developments Crockett and Chernoffmentioning

confidence: 99%

“…In theoretical investigations BFGS is known as a special case of the Broyden class [5]. Some Broyden updates with negative Broyden parameters have been found to produce faster convergence than BFGS updates [31], [8] but, for various reasons, have not been widely adopted. Indeed, Byrd et.…”

mentioning

confidence: 99%

See 4 more Smart Citations

Statistical Quasi-Newton: A New Look at Least Change

Liu¹,

Wiel²

2008

SIAM J. Optim.

View full text Add to dashboard Cite

Abstract.A new method for quasi-Newton minimization outperforms BFGS by combining least-change updates of the Hessian with step sizes estimated from a Wishart model of uncertainty. The Hessian update is in the Broyden family but uses a negative parameter, outside the convex range, that is usually regarded as the safe zone for Broyden updates. Although full Newton steps based on this update tend to be too long, excellent performance is obtained with shorter steps estimated from the Wishart model. In numerical comparisons to BFGS the new statistical quasi-Newton (SQN) algorithm typically converges with about 25% fewer iterations, functions, and gradient evaluations on the top 1/3 hardest unconstrained problems in the CUTE library. Typical improvement on the 1/3 easiest problems is about 5%. The framework used to derive SQN provides a simple way to understand differences among various Broyden updates such as BFGS and DFP and shows that these methods do not preserve accuracy of the Hessian, in a certain sense, while the new method does. In fact, BFGS, DFP, and all other updates with nonnegative Broyden parameters tend to inflate Hessian estimates, and this accounts for their observed propensity to correct eigenvalues that are too small more readily than eigenvalues that are too large. Numerical results on three new test functions validate these conclusions. Key words. BFGS, DFP, negative Broyden family, Wishart model AMS subject classifications. 65K10, 90C53DOI. 10.1137/040614700 1. Introduction. Quasi-Newton methods for unconstrained optimization are important computational tools in many scientific fields and are a standard subject in textbooks on computation. The BFGS method, proposed individually in [6], [14], [20], and [30], is implemented in most optimization software and is widely recognized as efficient. Generalizations of BFGS are available for large problems with memory limitations, for problems with bound constraints, and for a parallel computing environment. In theoretical investigations BFGS is known as a special case of the Broyden class [5]. Some Broyden updates with negative Broyden parameters have been found to produce faster convergence than BFGS updates [31], [8] but, for various reasons, have not been widely adopted. Indeed, Byrd et. al. conclude that "practical algorithms that preserve the excellent properties of the BFGS method are difficult to design." Nocedal and Wright [29] state that "the BFGS formula. . . is presently considered to be the most effective of all quasi-Newton updating formulae." In our opinion, BFGS remains the most popular front-runner because of two important unanswered

show abstract

Section: M-mentioning

confidence: 99%

Section: M-mentioning

confidence: 99%

Section: Historical Developments Crockett and Chernoffmentioning

confidence: 99%

“…Robust improvement over BFGS has been elusive. Indeed, Zhang and Tewarson [31] concluded that such investigations have not shaken the position of BFGS as the most popular front-runner.…”

Section: Historical Developments Crockett and Chernoffmentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

Statistical Quasi-Newton: A New Look at Least Change

Liu¹,

Wiel²

2008

SIAM J. Optim.

View full text Add to dashboard Cite

show abstract

Compact representation of the full Broyden class of quasi‐Newton updates

DeGuchy

Erway

Marcia

2018

Numerical Linear Algebra App

View full text Add to dashboard Cite

Summary In this paper, we present the compact representation for matrices belonging to the Broyden class of quasi‐Newton updates, where each update may be either rank one or rank two. This work extends previous results solely for the restricted Broyden class of rank‐two updates. In this article, it is not assumed that the same Broyden update is used in each iteration; rather, different members of the Broyden class may be used in each iteration. Numerical experiments suggest that a practical implementation of the compact representation is able to accurately represent matrices belonging to the Broyden class of updates. Furthermore, we demonstrate how to compute the compact representation for the inverse of these matrices and a practical algorithm for solving linear systems with members of the Broyden class of updates. We demonstrate through numerical experiments that the proposed linear solver is able to efficiently solve linear systems with members of the Broyden class of matrices with high accuracy. As an immediate consequence of this work, it is now possible to efficiently compute the eigenvalues of any limited‐memory member of the Broyden class of matrices, allowing for the computation of condition numbers and the ability to perform sensitivity analysis.

show abstract

An Overview of Unconstrained Optimization

Fletcher¹

1994

Algorithms for Continuous Optimization

View full text Add to dashboard Cite

Quasi-Newton Algorithms with Updates from the Preconvex Part of Broyden's Family

Cited by 46 publications

References 0 publications

Statistical Quasi-Newton: A New Look at Least Change

Statistical Quasi-Newton: A New Look at Least Change

Compact representation of the full Broyden class of quasi‐Newton updates

An Overview of Unconstrained Optimization

Contact Info

Product

Resources

About