Solving Local Minima Problem in Back Propagation Algorithm Using Adaptive Gain, Adaptive Momentum and Adaptive Learning Rate on Classification Problems

Abstract

This paper presents a new method to improve back propagation algorithm from getting stuck with local minima problem and slow convergence speeds which caused by neuron saturation in the hidden layer. In this proposed algorithm, each training pattern has its own activation functions of neurons in the hidden layer that are adjusted by the adaptation of gain parameters together with adaptive momentum and learning rate value during the learning process. The efficiency of the proposed algorithm is compared with the conventional back propagation gradient descent and the current working back propagation gradient descent with adaptive gain by means of simulation on three benchmark problems namely iris, glass and thyroid.

An Optimized Back Propagation Learning Algorithm with Adaptive Learning Rate

“…The author proposed a technique that adjusted the weights by the adaptation of gain parameters together with adaptive momentum and learning rate value during the learning process. The experiments results as presented in [16] demonstrated that the proposed method improved significantly the performance of the learning process.…”

“…On the other hand, if the learning rate is set to too small, the algorithm will take a long time to converge [12]. Many researches [13,14,15,16] used different strategies to speed up the convergence time by varying the learning rate. The best strategies in gradient descent BP is that it utilizes larger learning rate when the neural network model is far from the solution and smaller learning rate when the neural net is near the solution [17].…”

Maximum Entropy Markov Model for Human Activity Recognition Using Depth Camera

“…However, BPNN has two major limitations, being the low convergence rate and the instability [33]. These limitations are the results of being trapped in a local minimum [34], [35] and the possibility in overshooting the minimum of the error surface [33].…”

A Hybrid Simulated Annealing and Back-Propagation Algorithm for Feed forward Neural Network to Detect Credit Card Fraud

“…In the next step, besides the MSE, the network information back propagates from the output layer to the input layer, and specific connector weights are updated utilizing a "generalized ∆ rule" that is held of learning rate (η) and momentum constant (α) [11]. Equations 3 and 4 display the rule of weight updating.…”

“…For achieving the average η which gives the least MSE, rigorous parametric research has been conducted in this research [11]. The η at which the error is minimal is determined for the association with the SA algorithm.…”