Convergence mechanism of learning on backpropagation and high speed learning

Abstract

Neural Network (NN) are converted to output signal by 1/0 function such as the sigmoid function. The input signals at each layer unit is changed by the learning. Its change is depend on the slope of t h e 1/0 f u n c t i o n . I n t h e sensitive region as active region given by the 1/0 function, input signals move effectively and the movement increase with t h e i n c r e a s e of slope. A t t h e saturation region, however, it is very small. In this paper the limited linear function is used instead of the sigmoid function, because the active region is defined clearly and an examination of the movement easily. And the convergence mechanism of l e a r n i n g on Backpropagation is examined by the movement per a training cycle. Moreover, possibil i t y of t h e h i g h s p e e d l e a r n i n g is considered on the basis of its results. I NOn learning of the NN with Backpropagation, the output signals of units are calculated by the 1/0 function and they are used as new input signals at the next layer. This process is the same on it of the hidden layer and output layer. The output signals gotten at the o u t p u t l a y e r a r e compared with t h e s u p e r v i s e d s i g n a l s a n d t h e i n t e rconnecting weights are modified from the output layer to the input layer. This o p e r a t i o n is c o n t i n u e d u n t i l t h e difference between the output signals and the supervised signals are smaller than the minimum error. The modification of interconnecting weights decides the movement of input signals directly. In addition, 1/0 function is important factor for conversion of signals, because it decides the magnitude of movement of the input signals. This movement of input signals in sensitive region ("active region") is given by modification of interconnecting weights and 1/0 function. However its movement is s m a l l at the saturation region. For s t u d y i n g on c o n v e r g e n c e mechanism of l e a r n i n g , t h e sigmoid function which is used as 1/0 function is not suitable, because sigmoid function is non-linear function and the active region is not clear. Then limited linear function is used instead of sigmoid function. A s the limited linear function can define the active region and has a constant slope in t h e a c t i v e r e g i o n , t h e modeling of mechanism becomes simple and magnitudeof movement of input signals can be q u a n t i f i e d . I t is e x p e c t e d t h a t t h e increasing of movement in magnitude is effective mechanism of learning. And convergence mechanism becomes clear by the effect of the active region and the movement of t h e i n p u t s i g n a l s i n a training. .MethodThe modification of t h e i n t e rconnecting weights is given by Backpropagation and decides the movement of the input signals. Branch input signals (3) are led after each signal at units of the previous layer (Pj) are multiplied by the each interconnecting weight (wj). And the input signals (i) are given by summation of each branch input sQnals. ib:Branch input signal 1 8 1992 IEEE 523