Accelerating backpropagation using effective parameters at each step and an experimental evaluation

Abstract

An acceleration of backpropagation algorithm with momentum (BPM) is introduced. At every stage of the learning process, local quadratic approximation of the error function is performed and the Hessian matrix of the quadratic function is approximated. Effective learning rate and momentum factor are determined by means of maximum and minimum eigenvalues of the approximated Hessian matrix at each step. BPM algorithm is modified so as to work automatically with these effective parameters. Performance of this new approach is demonstrated in comparison with well-known training algorithms on conventional problems by an experimental evaluation.

Near optimal step size and momentum in gradient descent for quadratic functions

“…An application of this approach to a popular back-propagation algorithm in neural networks can be found in [7]. In this paper, training time of a multilayer neural network had been reduced significantly in various types of benchmark problems.…”