Some methods of speeding up the convergence of iteration methods

Polyak, B. T.

doi:10.1016/0041-5553(64)90137-5

Cited by 2,007 publications

(1,338 citation statements)

References 1 publication

Supporting

Mentioning

1,203

Contrasting

Unclassified

Order By: Relevance

“…Second-order methods also amplify steps in low-curvature directions, but instead of accumulating changes they reweight the update along each eigen-direction of the curvature matrix by the inverse of the associated curvature. And just as secondorder methods enjoy improved local convergence rates, Polyak (1964) showed that CM can considerably accelerate convergence to a local minimum, requiring p Rtimes fewer iterations than steepest descent to reach the same level of accuracy, where R is the condition number of the curvature at the minimum and µ is set to (…”

Section: Momentum and Nesterov's Accelerated Gradientmentioning

confidence: 99%

“…The momentum method (Polyak, 1964), which we refer to as classical momentum (CM), is a technique for accelerating gradient descent that accumulates a velocity vector in directions of persistent reduction in the objective across iterations. Given an objective function f (✓) to be minimized, classical momentum is given by:…”

Section: Momentum and Nesterov's Accelerated Gradientmentioning

confidence: 99%

See 1 more Smart Citation

Why Momentum Really Works.

Goh

2017

Distill

104

View full text Add to dashboard Cite

Deep and recurrent neural networks (DNNs and RNNs respectively) are powerful models that were considered to be almost impossible to train using stochastic gradient descent with momentum. In this paper, we show that when stochastic gradient descent with momentum uses a well-designed random initialization and a particular type of slowly increasing schedule for the momentum parameter, it can train both DNNs and RNNs (on datasets with long-term dependencies) to levels of performance that were previously achievable only with Hessian-Free optimization. We find that both the initialization and the momentum are crucial since poorly initialized networks cannot be trained with momentum and well-initialized networks perform markedly worse when the momentum is absent or poorly tuned.Our success training these models suggests that previous attempts to train deep and recurrent neural networks from random initializations have likely failed due to poor initialization schemes. Furthermore, carefully tuned momentum methods su ce for dealing with the curvature issues in deep and recurrent network training objectives without the need for sophisticated second-order methods.

show abstract

Section: Momentum and Nesterov's Accelerated Gradientmentioning

confidence: 99%

Section: Momentum and Nesterov's Accelerated Gradientmentioning

confidence: 99%

Why Momentum Really Works.

Goh

2017

Distill

104

View full text Add to dashboard Cite

show abstract

“…Polyak (1964), Rybashov (1969) and Tsypkin (1971)) proposed the use of dynamical systems to compute the solution of various optimization problems. Specifically, Polyak (1964) investigates the idea of using a dynamical system that represents a HBF, moving under Newtonian dynamics in a conservative force field. Later, a more detailed analysis of this system was carried out by Attouch et al (2000).…”

Section: Continuous Optimization Bpm and The Conjugate Gradient Algomentioning

confidence: 99%

Steepest descent with momentum for quadratic functions is a version of the conjugate gradient method

Bhaya

Kaszkurewicz

2004

Neural Networks

View full text Add to dashboard Cite

It is pointed out that the so called momentum method, much used in the neural network literature as an acceleration of the backpropagation method, is a stationary version of the conjugate gradient method. Connections with the continuous optimization method known as heavy ball with friction are also made. In both cases, adaptive (dynamic) choices of the so called learning rate and momentum parameters are obtained using a control Liapunov function analysis of the system. q

show abstract

“…To avoid overfitting, we used 2 regularization term [6] in both layers. Momentum method [13] was used when updating weights and biases. The following other hyper-parameters were selected for pretraining: learning rate in the first layer λ = 0.01, learning rate in the second layer λ = 0.001, initial momentum µ = 0.5, momentum after fifth epoch µ = 0.9, and weight penalty 2 = 2 · 10 −5 .…”

Section: Datasets and Experimentsmentioning

confidence: 99%

“…This work demonstrated that, under certain conditions, stochastic gradient descent can match the performance of Martens's Hessian-free optimizer. The conditions for these performance levels were the use of proper random initialization (which was SI in this case) and the use of momentum method [13] or Nesterov Accelerated Gradient [12] during training. Works cited above study various forms of random initialization for networks with no pretraining.…”

Section: Related Workmentioning

confidence: 99%

Effects of Sparse Initialization in Deep Belief Networks

Grzegorczyk

Kurdziel

Wójcik

2015

csci

View full text Add to dashboard Cite

Deep neural networks are often trained in two phases: first, hidden layers are pretrained in an unsupervised manner, and then the network is fine-tuned with error backpropagation. Pretraining is often carried out using Deep Belief Networks (DBNs), with initial weights set to small random values. However, recent results established that well-designed initialization schemes, e.g., Sparse Initialization (SI), can greatly improve the performance of networks that do not use pretraining. An interesting question arising from these results is whether such initialization techniques wouldn't also improve pretrained networks. To shed light on this question, in this work we evaluate SI in DBNs that are used to pretrain discriminative networks. The motivation behind this research is our observation that SI has an impact on the features learned by a DBN during pretraining. Our results demonstrate that this improves network performance: when pretraining starts from sparsely initialized weight matrices, networks achieve lower classification errors after fine-tuning.

show abstract

Some methods of speeding up the convergence of iteration methods

Cited by 2,007 publications

References 1 publication

Why Momentum Really Works.

Why Momentum Really Works.

Steepest descent with momentum for quadratic functions is a version of the conjugate gradient method

Effects of Sparse Initialization in Deep Belief Networks

Contact Info

Product

Resources

About