A first order adaptive learning rate algorithm for backpropagation networks

Nachtsheim, P. R.

doi:10.1109/icnn.1994.374171

Cited by 4 publications

(4 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Obtained results regarding the stability and the faster convergence of algorithm (1) can play the key role in determining the effective η and μ at every stage; that is, in the case ofx = x t happening at every stage t. Performing the local quadratic approximation of the error function (16), effective η 0 t and μ 0 t coefficients can be calculated by formulas (10) and (11) using the smallest and the largest eigenvalues of the Hessian matrix (17) at every stage. This idea provides the dynamic selection of effective η t and μ t parameters in algorithm (15), and this fairly affects the convergence speed of the algorithm, as the experimental results demonstrate in section 4.…”

Section: Accelerated Backpropagation With Effective Parametersmentioning

confidence: 90%

Accelerating backpropagation using effective parameters at each step and an experimental evaluation

Mammadov

Tas

Omay

2008

Journal of Statistical Computation and Simulation

View full text Add to dashboard Cite

An acceleration of backpropagation algorithm with momentum (BPM) is introduced. At every stage of the learning process, local quadratic approximation of the error function is performed and the Hessian matrix of the quadratic function is approximated. Effective learning rate and momentum factor are determined by means of maximum and minimum eigenvalues of the approximated Hessian matrix at each step. BPM algorithm is modified so as to work automatically with these effective parameters. Performance of this new approach is demonstrated in comparison with well-known training algorithms on conventional problems by an experimental evaluation.

show abstract

Section: Accelerated Backpropagation With Effective Parametersmentioning

confidence: 90%

Accelerating backpropagation using effective parameters at each step and an experimental evaluation

Mammadov

Tas

Omay

2008

Journal of Statistical Computation and Simulation

View full text Add to dashboard Cite

show abstract

“…Second order training algorithms are of different class. Other derivative based ANN application-training algorithms are discussed in Wang and Lin (1998), Van Ooyen and Nienhuis (1992), Towsey et al (1995), Subramanian and Hung (1990) and Nachtsheim (1994). Dissimilarity should therefore be made to exhibit the computational complexity between derivative free, first and second order derivative based training methods.…”

Section: Derivative Based Training Methodsmentioning

confidence: 99%

Degenerated simplex search method to optimize neural network error function

Ahmed

2013

Kybernetes

View full text Add to dashboard Cite

Purpose -The purpose of this paper is to present a degenerated simplex search method to optimize neural network error function. By repeatedly reflecting and expanding a simplex, the centroid property of the simplex changes the location of the simplex vertices. The proposed algorithm selects the location of the centroid of a simplex as the possible minimum point of an artificial neural network (ANN) error function. The algorithm continually changes the shape of the simplex to move multiple directions in error function space. Each movement of the simplex in search space generates local minimum. Simulating the simplex geometry, the algorithm generates random vertices to train ANN error function. It is easy to solve problems in lower dimension. The algorithm is reliable and locates minimum function value at the early stage of training. It is appropriate for classification, forecasting and optimization problems. Design/methodology/approach -Adding more neurons in ANN structure, the terrain of the error function becomes complex and the Hessian matrix of the error function tends to be positive semi-definite. As a result, derivative based training method faces convergence difficulty. If the error function contains several local minimum or if the error surface is almost flat, then the algorithm faces convergence difficulty. The proposed algorithm is an alternate method in such case. This paper presents a non-degenerate simplex training algorithm. It improves convergence by maintaining irregular shape of the simplex geometry during degenerated stage. A randomized simplex geometry is introduced to maintain irregular contour of a degenerated simplex during training. Findings -Simulation results show that the new search is efficient and improves the function convergence. Classification and statistical time series problems in higher dimensions are solved. Experimental results show that the new algorithm (degenerated simplex algorithm, DSA) works better than the random simplex algorithm (RSM) and back propagation training method (BPM). Experimental results confirm algorithm's robust performance.Research limitations/implications -The algorithm is expected to face convergence complexity for optimization problems in higher dimensions. Good quality suboptimal solution is available at the early stage of training and the locally optimized function value is not far off the global optimal solution, determined by the algorithm. Practical implications -Traditional simplex faces convergence difficulty to train ANN error function since during training simplex can't maintain irregular shape to avoid degeneracy. Simplex size becomes extremely small. Hence convergence difficulty is common. Steps are taken to redefine simplex so that the algorithm avoids the local minimum. The proposed ANN training method is derivative free. There is no demand for first order or second order derivative information hence making it simple to train ANN error function. Originality/value -The algorithm optimizes ANN error function, when the Hessian matrix of error ...

show abstract

“…The positive constant of , which is selected by user, is called the learning rate, where (0,1). b. Dynamically adjusting the learning rate, either commonly for all weights [21], [26] or separately for each weight [24]. (2) Dynamical adaptation of the weight adjustments expressed by the vector k d i.e.…”

Section: ………………………………………………………………(2)mentioning

confidence: 99%

“…Consider any method of the form (4) and (5) where the learning rate k  satisfies the standard Wolfe conditions equation (9) and (10) and the search direction computed by the equation (21) and (22), then for…”

Section: Theoremmentioning

confidence: 99%

Conjugate Gradient Algorithm Based on Aitken's Process for Training Neural Networks

Abbo¹,

Mohammed²

2014

AL-Rafidain Journal of Computer Sciences and Mathematics

View full text Add to dashboard Cite

Conjugate gradient methods constitute excellent neural network training methods, because of their simplicity, numerical efficiency and their very low memory requirements. It is wellknown that the procedure of training a neural network is highly consistent with unconstrained optimization theory and many attempts have been made to speed up this process. In particular, various algorithms motivated from numerical optimization theory have been applied for accelerating neural network training. In this paper, we propose a conjugate gradient neural network training algorithm by using Aitken's process which guarantees sufficient descent with Wolfe line search. Moreover, we establish that our proposed method is globally convergent for general functions under the strong Wolfe conditions. In the experimental results, we compared the behavior of our proposed method(NACG) with well-known methods in this field.

show abstract

A first order adaptive learning rate algorithm for backpropagation networks

Cited by 4 publications

References 1 publication

Accelerating backpropagation using effective parameters at each step and an experimental evaluation

Accelerating backpropagation using effective parameters at each step and an experimental evaluation

Degenerated simplex search method to optimize neural network error function

Conjugate Gradient Algorithm Based on Aitken's Process for Training Neural Networks

Contact Info

Product

Resources

About