Representations of quasi-Newton matrices and their use in limited memory methods

An interior-point method for nonlinear programming is presented. It enjoys the flexibility of switching between a line search method that computes steps by factoring the primal-dual equations and a trust region method that uses a conjugate gradient iteration. Steps computed by direct factorization are always tried first, but if they are deemed ineffective, a trust region iteration that guarantees progress toward stationarity is invoked. To demonstrate its effectiveness, the algorithm is implemented in the Knitro [6,28] software package and is extensively tested on a wide selection of test problems.

show abstract

“…We have implemented a limited memory BFGS method using the compact representations described in [8]. Here B has the form…”

Section: Quasi-newton Approximationsmentioning

confidence: 99%

An interior algorithm for nonlinear optimization that combines line search and trust region steps

Waltz¹,

Morales

Nocedal³

et al. 2005

Math. Program.

883

425

View full text Add to dashboard Cite

show abstract

“…the Quasi Newton Algorithm (QNA; Byrd et al 1994) and the Levenberg-Marquardt rule (Nocedal and Wright 2006), respectively.…”

Section: The Machine Learning Modelsmentioning

confidence: 99%

“…a MLP implementation with learning rule based on the Quasi Newton Algorithm, belongs to the Newton's methods specialized to find the stationary point of a function through a statistical approximation of the Hessian of the training error, obtained by a cyclic gradient calculation. MLPQNA makes use of the known L-BFGS algorithm (Limited memory -Broyden Fletcher Goldfarb Shanno; Byrd et al 1994), originally designed for problems with a wide parameter space. The analytical details of the MLPQNA method, as well as its performances, have been extensively discussed elsewhere (Cavuoti et al 2012;Brescia et al 2013;Cavuoti et al 2014Cavuoti et al , 2015b.…”

Section: The Machine Learning Modelsmentioning

confidence: 99%

A cooperative approach among methods for photometric redshifts estimation: an application to KiDS data

Cavuoti

Tortora

Brescia

et al. 2016

Mon. Not. R. Astron. Soc.

View full text Add to dashboard Cite

Photometric redshifts (photo-z's) are fundamental in galaxy surveys to address different topics, from gravitational lensing and dark matter distribution to galaxy evolution. The Kilo Degree Survey (KiDS), i.e. the ESO public survey on the VLT Survey Telescope (VST), provides the unprecedented opportunity to exploit a large galaxy dataset with an exceptional image quality and depth in the optical wavebands. Using a KiDS subset of about 25, 000 galaxies with measured spectroscopic redshifts, we have derived photo-z's using i) three different empirical methods based on supervised machine learning, ii) the Bayesian Photometric Redshift model (or BPZ), and iii) a classical SED template fitting procedure (Le PHARE). We confirm that, in the regions of the photometric parameter space properly sampled by the spectroscopic templates, machine learning methods provide better redshift estimates, with a lower scatter and a smaller fraction of outliers. SED fitting techniques, however, provide useful information on the galaxy spectral type which can be effectively used to constrain systematic errors and to better characterize potential catastrophic outliers. Such classification is then used to specialize the training of regression machine learning models, by demonstrating that a hybrid approach, involving SED fitting and machine learning in a single collaborative framework, can be effectively used to improve the accuracy of photo-z estimates.

show abstract

“…This is achieved with the use of the following recursive formula which is directly derived from (22) [8].…”

Section: Projected Gradient Non-negative Least-squares Algorithmmentioning

confidence: 99%

Fast Non-Negative Least-Squares Learning in the Random Neural Network

Timotheou

2016

Prob. Eng. Inf. Sci.

View full text Add to dashboard Cite

The random neural network is a biologically inspired neural model where neurons interact by probabilistically exchanging positive and negative unit-amplitude signals that has superior learning capabilities compared to other artificial neural networks. This paper considers non-negative least squares supervised learning in this context, and develops an approach that achieves fast execution and excellent learning capacity. This speedup is a result of significant enhancements in the solution of the non-negative least-squares problem which regard (a) the development of analytical expressions for the evaluation of the gradient and objective functions and (b) a novel limited-memory quasi-Newton solution algorithm. Simulation results in the context of optimizing the performance of a disaster management problem using supervised learning verify the efficiency of the approach, achieving two orders of magnitude execution speedup and improved solution quality compared to state-of-the-art algorithms.

show abstract

Representations of quasi-Newton matrices and their use in limited memory methods

Cited by 540 publications

References 13 publications

An interior algorithm for nonlinear optimization that combines line search and trust region steps

An interior algorithm for nonlinear optimization that combines line search and trust region steps

A cooperative approach among methods for photometric redshifts estimation: an application to KiDS data

Fast Non-Negative Least-Squares Learning in the Random Neural Network

Contact Info

Product

Resources

About