Steepest descent with momentum for quadratic functions is a version of the conjugate gradient method

Bhaya, Amit; Kaszkurewicz, E.

doi:10.1016/s0893-6080(03)00170-9

Cited by 75 publications

(38 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This has been exploited in tracking applications in the past [26]. While this simple ODE maps to the stochastic gradient update, higher order ODEs can be shown to map to more sophisticated updates, such as gradient descent with momentum [28]. These SA algorithms with momentum have also been analyzed to some extent in [29].…”

Section: Multiplying Both Sides Of This Equation Bymentioning

confidence: 99%

A tutorial on stochastic approximation algorithms for training Restricted Boltzmann Machines and Deep Belief Nets

Swersky

Chen

Marlin

et al. 2010

2010 Information Theory and Applications Workshop (ITA)

View full text Add to dashboard Cite

Abstract-In this study, we provide a direct comparison of the Stochastic Maximum Likelihood algorithm and Contrastive Divergence for training Restricted Boltzmann Machines using the MNIST data set. We demonstrate that Stochastic Maximum Likelihood is superior when using the Restricted Boltzmann Machine as a classifier, and that the algorithm can be greatly improved using the technique of iterate averaging from the field of stochastic approximation. We further show that training with optimal parameters for classification does not necessarily lead to optimal results when Restricted Boltzmann Machines are stacked to form a Deep Belief Network. In our experiments we observe that fine tuning a Deep Belief Network significantly changes the distribution of the latent data, even though the parameter changes are negligible.

show abstract

Section: Multiplying Both Sides Of This Equation Bymentioning

confidence: 99%

A tutorial on stochastic approximation algorithms for training Restricted Boltzmann Machines and Deep Belief Nets

Swersky

Chen

Marlin

et al. 2010

2010 Information Theory and Applications Workshop (ITA)

View full text Add to dashboard Cite

show abstract

“…This change is better for the behavior of the algorithm since it provides the chance of escaping surface local minimums. Of course the better choice of a and l constants speed up convergence of the algorithm [2,24,32].…”

Section: Standard Backpropagation (Bp) Training Algorithmmentioning

confidence: 99%

Analysis of international debt problem using artificial neural networks and statistical methods

Yazıcı

Memmedli

Aslanargun

et al. 2010

Neural Comput & Applic

View full text Add to dashboard Cite

It is known from the scientific researches that artificial neural networks are alternatives of statistical methods such as regression analysis and classification in recent years. Since multi-layer backpropagation neural network models are nonlinear, it is expected that the neural network models should make better classifications and predictions. The studies on this subject support that idea. In this study, a macro-economic problem on rescheduling or non-rescheduling of the countries' international debts is taken into account. Among the statistical methods, logistic and probit regression, and the different neural network backpropagation algorithms are applied and comparisons are made. Evaluations and suggestions are made depending on the results and different neural network architecture.

show abstract

“…The idea of momentum acceleration comes from neural network algorithms and it is introduced by Rumelhart, Hinton and Williams [16]. Many researchers have developed the theory about momentum and extended its applications, see, e.g., [17][18][19][20][21][22][23]. Here we point out that N. Qian studies its mechanisms.…”

Section: Introductionmentioning

confidence: 99%

On parameter acceleration methods for saddle point problems

Zhang

2015

Journal of Computational and Applied Mathematics

View full text Add to dashboard Cite

a b s t r a c tFor solving saddle point problems, parameter acceleration methods which include Uzawatype methods are investigated by many researchers in the literature. In this paper, we introduce the inexact Uzawa method with another parameter acceleration, that is, the socalled momentum acceleration method for solving saddle point problems. We discuss the convergence conditions of the inexact Uzawa iteration with momentum acceleration and give the optimal momentum factors which minimize the spectral radii of the associated iteration matrices. Numerical results demonstrate the effectiveness of the inexact Uzawa method with momentum acceleration and the mixed parameter acceleration methods for solving saddle point problems.

show abstract

Steepest descent with momentum for quadratic functions is a version of the conjugate gradient method

Cited by 75 publications

References 16 publications

A tutorial on stochastic approximation algorithms for training Restricted Boltzmann Machines and Deep Belief Nets

A tutorial on stochastic approximation algorithms for training Restricted Boltzmann Machines and Deep Belief Nets

Analysis of international debt problem using artificial neural networks and statistical methods

On parameter acceleration methods for saddle point problems

Contact Info

Product

Resources

About