Deterministic Convergence of an Online Gradient Method for BP Neural Networks

Wu, Wei; Feng, Guorui; Li, Zhengxue; Xu, Yuesheng

doi:10.1109/tnn.2005.844903

Cited by 139 publications

(52 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The number of layers of perceptrons is made up of the number of layers in a NN. FFNNs could be used to map any function from input to output and they are known as gradient based learning algorithms (Steepest Decent Method) which is the supreme algorithm used in FFNNs [6][7][8][9][10]. However, since FFNNs model is based on human thought processes, it is essential to appreciate how the human brain functions on a basic level [11,12].…”

Section: Taxonomy Of Fnnmentioning

confidence: 99%

“…More so, there's no need to use more than one HL for many real-world issues and problems that require two HLs are seldom bump into. Differences between the numbers of HLs are summarized in table 1 below: [7] Again, deciding on the number of hidden neurons in layers is a very critical part in determining the overall neural network architecture. Although they do not directly relate with the external environment these layers have an incredible impact on the ultimate output.…”

Section: Fig 4 Conduit Structure Of Ffnnmentioning

confidence: 99%

See 1 more Smart Citation

Taxonomy and a Theoretical Model for Feedforward Neural Networks

Benuwa¹,

Zhan²,

Ghansah³

et al. 2017

IJCA

View full text Add to dashboard Cite

Feedforward Neural Network (FFNN) is a surrogate of Artificial Neural Network (ANN) in which links amongst the units do not form a directed cycle. ANNs, akin to the vast network of neurons in the brain (human central nervous system) are usually presented as systems of interweaving connected "neurons" which exchange messages between each other. These connections have numeric hefts that can be adjusted and grounded on experience, enforcing adaptively on neural networks to inputs and learning capabilities. This paper presents a comprehensive review of FFNN with emphasis on implantation issues, which have been addressed by previous approaches. We also propose a theoretical model that exhibits potential superior performances in terms of convergence speed, efficient and effective computation and generality than state of the art models.

show abstract

Section: Taxonomy Of Fnnmentioning

confidence: 99%

Section: Fig 4 Conduit Structure Of Ffnnmentioning

confidence: 99%

Taxonomy and a Theoretical Model for Feedforward Neural Networks

Benuwa¹,

Zhan²,

Ghansah³

et al. 2017

IJCA

View full text Add to dashboard Cite

show abstract

“…On the other hand, the fact that { } n V is monotonically non-increasing sequence is not necessary to achieve (20) in principle.…”

Section: Preliminariesmentioning

confidence: 99%

“…Unfortunately, this gives that the learning goes faster in the beginning and slows down in the late stage. The convergence analysis of learning algorithm with deterministic (non-stochastic) nature has been given in [17][18][19][20][21][22]. In contrast to the stochastic approach, several of these results allow to employ a constant learning rate [19,23].…”

Section: Introductionmentioning

confidence: 99%

Asymptotic Behaviour of Gradient Learning Algorithms in Neural Network Models for the Identification of Nonlinear Systems

Azarskov¹,

Kucherov²,

Nikolaienko³

et al. 2015

AJNNA

View full text Add to dashboard Cite

This paper deals with studying the asymptotical properties of multilayer neural networks models used for the adaptive identification of wide class of nonlinearly parameterized systems in stochastic environment. To adjust the neural network's weights, the standard online gradient type learning algorithms are employed. The learning set is assumed to be infinite but bounded. The Lyapunov-like tool is utilized to analyze the ultimate behaviour of learning processes in the presence of stochastic input variables. New sufficient conditions guaranteeing the global convergence of these algorithms in the stochastic frameworks are derived. The main their feature is that they need no a penalty term to achieve the boundedness of weight sequence. To demonstrate asymptotic behaviour of the learning algorithms and support the theoretical studies, some simulation examples are also given.

show abstract

“…These problems are inherent to the basic learning rule of FNN that are based on GD optimization methods [15,30]. The convergence properties of such algorithms are discussed in [5,7,12,15,16,18], and [22]. Learning algorithms based on GD includes real-time recurrent learning (RTRL), ordered derivative learning and so on [1].…”

Section: Introductionmentioning

confidence: 99%

Designing stable neural identifier based on Lyapunov method

Alibakhshi

Teshnehlab

Alibakhshi

et al. 2015

JAIDM

View full text Add to dashboard Cite

The stability of learning rate in neural network identifiers and controllers is one of the challenging issues, which attract many researchers' interest in neural networks. This paper suggests adaptive gradient descent algorithm with stable learning laws for modified dynamic neural network (MDNN) and studies the stability of this algorithm. Also, stable learning algorithm for parameters of MDNN is proposed. By the proposed method, some constraints are obtained for learning rate. Lyapunov stability theory is applied to study the stability of the proposed algorithm. The Lyapunov stability theory guaranteed the stability of the learning algorithm. In the proposed method, the learning rate can be calculated online and will provide an adaptive learning rate for the MDNN structure. Simulation results are given to validate the results.

show abstract

Deterministic Convergence of an Online Gradient Method for BP Neural Networks

Cited by 139 publications

References 17 publications

Taxonomy and a Theoretical Model for Feedforward Neural Networks

Taxonomy and a Theoretical Model for Feedforward Neural Networks

Asymptotic Behaviour of Gradient Learning Algorithms in Neural Network Models for the Identification of Nonlinear Systems

Designing stable neural identifier based on Lyapunov method

Contact Info

Product

Resources

About