A Numerical Study of the Limited Memory BFGS Method and the Truncated-Newton Method for Large Scale Optimization

Nash, Stephen G.; Nocedal, Jorge

doi:10.1137/0801023

Cited by 180 publications

(120 citation statements)

References 15 publications

Supporting

Mentioning

117

Contrasting

Order By: Relevance

“…Therefore we may classify our test problem as a highly nonlinear problem near the solution. Then it is not surprising that the LBFGS algorithm outperforms the truncated-Newton algorithm in terms of CPU time (Table 2) if we take into account Nash's observation that for most of highly nonlinear problems, the LBFGS algorithm performs better than the truncated-Newton algorithm [32]. Our point is that even in this case, if we use a more accurate line search direction in the truncated-Newton algorithm, the truncated-Newton algorithm can outperform the LBFGS as the adj oint truncated-Newton algorithm does for the particular optimal control problem tested here.…”

Section: Numerical Resultsmentioning

confidence: 99%

“…where U is a point between the starting point U0 and the solution U*, p = 0 -U* and pt denotes the transpose of p. DN(U) gives a measure of the size of the third derivative or a deviation from quadratic behavior (see [32]). …”

Section: Numerical Resultsmentioning

confidence: 99%

“…We provide a brief description of the adjoint truncated-Newton algorithm which differs from the truncated-Newton algorithm ( [29], [32]) only in the Hessian vector calculation required for solving the Newton equations at the k-th iteration ck~ = -~k,…”

Section: The Adjoint Truncated-newton Methodsmentioning

confidence: 99%

“…The truncated-Newton algorithms attempt to blend the rapid (quadratic) convergence rate of classic Newton method with feasible storage [29] and computational requirements [32] for large-scale unconstrained minimization problems. When these algorithms are used for the large-scale unconstrained minimization [53], the Hessian vector product is usually obtained by applying a finite-difference approximation technique to the gradient of the cost function with respect to the initial conditions, while the gradient is calculated by using the first order adjoint technique.…”

Section: Introductionmentioning

confidence: 99%

“…Among the feasible methods for large-scale unconstrained minimization are (a) the limited memory conjugate gradient method ( [34], [43]); (b) quasi-Newton type algorithms ( [5], [9], [14], [37]); (c) limited-memory quasi-Newton methods such as LBFGS algorithm ( [23], [36]) and (d)truncated-Newton algorithms ( [26], [28], [29], [32], [411, [42]). …”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

A truncated Newton optimization algorithm in meteorology applications with analytic Hessian/vector products

Wang

Navon

Zou

et al. 1995

Comput Optim Applic

View full text Add to dashboard Cite

Abstract.A modified version of the truncated-Newton algorithm of Nash ([24], 125], [29]) is presented differing from it only in the use of an exact Hessian vector product for carrying out the large-scale unconstrained optimization required in variational data assimilation. The exact Hessian vector product is obtained by solving an optimal control problem of distributed parameters. (i.e. the system under study occupies a certain spatial and temporal domain and is modeled by partial differential equations) The algorithm is referred to as the adjoint truncated-Newton algorithm. The adjoint truncated-Newton algorithm is based on the first and the second order adjoint techniques allowing to obtain a better approximation to the Newton line search direction for the problem tested here. The adjoint truncated-Newton algorithm is applied here to a limited-area shallow water equations model with model generated data where the initial conditions serve as control variables. We compare the performance of the adjoint truncated-Newton algorithm with that of the original truncated-Newton method [29] and the LBFGS (Limited Memory BFGS) method of Liu and Nocedal [23]. Our numerical tests yield results which are twice as fast as these obtained by the truncated-Newton algorithm and are faster than the LBFGS method both in terms of number of iterations as well as in terms of CPU time.

show abstract

Section: Numerical Resultsmentioning

confidence: 99%

Section: Numerical Resultsmentioning

confidence: 99%

Section: The Adjoint Truncated-newton Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

A truncated Newton optimization algorithm in meteorology applications with analytic Hessian/vector products

Wang

Navon

Zou

et al. 1995

Comput Optim Applic

View full text Add to dashboard Cite

show abstract

Performance of efficient minimization algorithms as applied to models of peptides and proteins

1999

View full text Add to dashboard Cite

We test the efficiency of three minimization algorithms as applied to models of peptides and proteins. These include: the limited memory quasi‐Newton (L‐BFGS) of Liu and Nocedal; the truncated Newton (TN) with automatic preconditioner of Nash; and the nonlinear conjugate gradients (CG) of Shanno and Phua. The molecules are modeled by two energy functions, one is the Gromos87 united atoms force field (defining the energy EGRO), which takes into account the intramolecular interactions only; the second is defined by the energy Etot=EGRO+Esolv, where Esolv is an implicit solvation free every term based on the solvent‐accessible surface area of the atoms. The molecules studied are cyclo‐(d‐Pro1–Ala2–Ala3–Ala4–Ala5) (31 atoms), axinastatin 2 [cyclo‐(Asn1–Pro2–Phe3–Val4–Leu5–Pro6–Val7), 62 atoms], and the protein bovine pancreatic trypsin inhibitor (58 residues, 568 atoms). With EGRO, the performance of TN with respect to the CPU time is found to be ∼1.2 to 2 times better than that of both L‐BFGS and CG, whereas, with Etot, L‐BFGS outperforms TN by a factor of 1.5 to 2.5, and CG by a larger factor. Still, the quality of the solution in terms of the value of the minimized energy and the gradient norm, obtained with TN, is always equivalent to, or better than, those obtained with L‐BFGS and CG. The performance is analyzed in terms of criteria outlined by Nash and Nocedal. We find the distribution of the Hessian eigenvalues to be a reliable predictor of efficiency. ©1999 John Wiley & Sons, Inc. J Comput Chem 20: 354–364, 1999

show abstract

Geometry Optimization: 2

Schlick

1998

Encyclopedia of Computational Chemistry

View full text Add to dashboard Cite

A Numerical Study of the Limited Memory BFGS Method and the Truncated-Newton Method for Large Scale Optimization

Cited by 180 publications

References 15 publications

A truncated Newton optimization algorithm in meteorology applications with analytic Hessian/vector products

A truncated Newton optimization algorithm in meteorology applications with analytic Hessian/vector products

Performance of efficient minimization algorithms as applied to models of peptides and proteins

Geometry Optimization: 2

Contact Info

Product

Resources

About