Conjugate Gradient Algorithm Based on Aitken's Process for Training Neural Networks

Abstract

Conjugate gradient methods constitute excellent neural network training methods, because of their simplicity, numerical efficiency and their very low memory requirements. It is wellknown that the procedure of training a neural network is highly consistent with unconstrained optimization theory and many attempts have been made to speed up this process. In particular, various algorithms motivated from numerical optimization theory have been applied for accelerating neural network training. In this paper, we propose a conjugate gradient neural network training algorithm by using Aitken's process which guarantees sufficient descent with Wolfe line search. Moreover, we establish that our proposed method is globally convergent for general functions under the strong Wolfe conditions. In the experimental results, we compared the behavior of our proposed method(NACG) with well-known methods in this field.

New Scaled Conjugate Gradient Algorithm for Training Artificial Neural Networks Based on Pure Conjugacy Condition

“…Abbo and Mohammed in [6] suggested a new CG algorithm NA  based on the Aitken's process, in this section we try to generalize the method to more general form known as scaled conjugate gradient methods. Consider the search direction of the form :…”

“…Consider the N1SCG method where the learning rate k  satisfies the standard Wolfe conditions equation (6) and 7and if…”

“…x is the output of the j-th neuron which belongs to the l -th layer, ) ( f is the activation function and l j O is the output of the nod j at the output layer. Recently many learning algorithms for feed-forward neural networks has been discovered [4,5,6]. Several of these algorithms are based on a known method in optimization theory known as the gradient descent algorithm.…”