The conjugate gradient method in extremal problems

Polyak, B. T.

doi:10.1016/0041-5553(69)90035-4

Cited by 967 publications

(458 citation statements)

References 5 publications

Supporting

Mentioning

407

Contrasting

Unclassified

Order By: Relevance

“…One could employ iterative shrinkage [12] or the gradient projection method [24,5] for the prediction phase; in this paper we chose a special form of the latter. For the subspace phase, one could minimize F S [17,9] instead of a quadratic model of this function [8,20], but we choose to work with a model due to the high cost of evaluating the objective function.…”

Section: A Newton-cg Methods For L 1 Regularized Modelsmentioning

confidence: 99%

Sample size selection in optimization methods for machine learning

et al. 2012

View full text Add to dashboard Cite

This paper presents a methodology for using varying sample sizes in batch-type optimization methods for large scale machine learning problems. The first part of the paper deals with the delicate issue of dynamic sample selection in the evaluation of the function and gradient. We propose a criterion for increasing the sample size based on variance estimates obtained during the computation of a batch gradient. We establish an O(1/ ) complexity bound on the total cost of a gradient method. The second part of the paper describes a practical Newton method that uses a smaller sample to compute Hessian vector-products than to evaluate the function and the gradient, and that also employs a dynamic sampling technique. The focus of the paper shifts in the third part of the paper to L 1 regularized problems designed to produce sparse solutions. We propose a Newton-like method that consists of two phases: a (minimalistic) gradient projection phase that identifies zero variables, and subspace phase that applies a subsampled Hessian Newton iteration in the free variables. Numerical tests on speech recognition problems illustrate the performance of the algorithms.

show abstract

Section: A Newton-cg Methods For L 1 Regularized Modelsmentioning

confidence: 99%

Sample size selection in optimization methods for machine learning

et al. 2012

View full text Add to dashboard Cite

show abstract

“…The tangential subproblem (3.12) in the Interior/CG algorithm and the EQP phase (4.6) of the Active algorithm both require the solution of an equality constrained quadratic program. We solve these problems using a projected conjugate gradient iteration [10,20,24,26,32], which is well suited for large problems and can handle the negative curvature case without the need for Hessian modifications. We now outline this iteration and refer the reader to [20] for a more detailed derivation.…”

Section: Projected Cg Iterationmentioning

confidence: 99%

Knitro: An Integrated Package for Nonlinear Optimization

Byrd

Nocedal

Waltz

2006

Nonconvex Optimization and Its Applications

746

518

View full text Add to dashboard Cite

This paper describes Knitro 5.0, a C-package for nonlinear optimization that combines complementary approaches to nonlinear optimization to achieve robust performance over a wide range of application requirements. The package is designed for solving large-scale, smooth nonlinear programming problems, and it is also effective for the following special cases: unconstrained optimization, nonlinear systems of equations, least squares, and linear and quadratic programming. Various algorithmic options are available, including two interior methods and an active-set method. The package provides crossover techniques between algorithmic options as well as automatic selection of options and settings.

show abstract

“…Hence, (23) and (24) hold, which implies that Q k+1 is a symmetric positive definite matrix when σ (y T k s k ) > 0.…”

Section: Theorem 2 Let λ (K+1)mentioning

confidence: 99%

“…The different choices for the parameter β k correspond to different CG methods, such as HS method [15], FR method [7], PRP method [22,23], LS method [16], PRP + method [8], DY method [5] and so on. On the history of the conjugate gradient method, there are several survey articles, such as [11].…”

Section: Introductionmentioning

confidence: 99%

Symmetric Perry conjugate gradient method

Liu

2013

Comput Optim Appl

View full text Add to dashboard Cite

A family of new conjugate gradient methods is proposed based on Perry's idea, which satisfies the descent property or the sufficient descent property for any line search. In addition, based on the scaling technology and the restarting strategy, a family of scaling symmetric Perry conjugate gradient methods with restarting procedures is presented. The memoryless BFGS method and the SCALCG method are the special forms of the two families of new methods, respectively. Moreover, several concrete new algorithms are suggested. Under Wolfe line searches, the global convergence of the two families of the new methods is proven by the spectral analysis for uniformly convex functions and nonconvex functions. The preliminary numerical comparisons with CG_DESCENT and SCALCG algorithms show that these new algorithms are very effective algorithms for the large-scale unconstrained optimization problems. Finally, a remark for further research is suggested.

show abstract

The conjugate gradient method in extremal problems

Cited by 967 publications

References 5 publications

Sample size selection in optimization methods for machine learning

Sample size selection in optimization methods for machine learning

Knitro: An Integrated Package for Nonlinear Optimization

Symmetric Perry conjugate gradient method

Contact Info

Product

Resources

About