Simultaneous Perturbation Learning Rule for Recurrent Neural Networks and Its FPGA Implementation

Abstract

Recurrent neural networks have interesting properties and can handle dynamic information processing unlike ordinary feedforward neural networks. However, they are generally difficult to use because there is no convenient learning scheme. In this paper, a recursive learning scheme for recurrent neural networks using the simultaneous perturbation method is described. The detailed procedure of the scheme for recurrent neural networks is explained. Unlike ordinary correlation learning, this method is applicable to analog learning and the learning of oscillatory solutions of recurrent neural networks. Moreover, as a typical example of recurrent neural networks, we consider the hardware implementation of Hopfield neural networks using a field-programmable gate array (FPGA). The details of the implementation are described. Two examples of a Hopfield neural network system for analog and oscillatory targets are shown. These results show that the learning scheme proposed here is feasible.

Noise removal in electroencephalogram signals using an artificial neural network based on the simultaneous perturbation method

“…At all these stages, the ANN was adapted using the SP method in order to obtain the best results. The process of training and initialization was modified and implemented as described by Maeda [50][51][52] (Fig. 2).…”

“…Therefore, it is relatively easy to implement as a learning rule of ANNs compared with other learning rules such as the back propagation (BP) learning rule. At the same time, this learning rule is easily applicable to recurrent types of ANNs, since only the final error values are required to estimate the gradient of the error function with respect to the weights [50][51][52]. The ANN has been also chosen mainly because of its adaptability to the nonlinear and time-varying features of the noise.…”

Simultaneous Perturbation Particle Swarm Optimization and Its FPGA Implementation

“…J.AIespector et al and G.Cauwenberghs also individually proposed a parallel gradient descent method and stochastic error descent algorithm, respectively, which are identical to the simultaneous perturbation learning rule (Cauwenberghs, 1993) (Alespector et al, 1993). Many applications of the simultaneous perturbation are reported in the fields of neural networks (Maeda, 1997) and their hardware implementation (Maeda, 2003) (Maeda, 2005). The simultaneous perturbation method is described as follows;…”

A convergent algorithm for a cascade network of multiplexed dual outputdiscrete perceptrons for linearly nonseparable classification

“…As a method for implementation, the FPGA approach, which uses reprogrammable digital ICs, is chosen since the usage of FPGA for neural network implementation provides flexibility of programmable systems along with the power and speed of parallel hardware architectures [10,11,12,13,14,15,16,17].…”