Learning processes in neural networks

Heskes, Tom; Kappen, Bert

doi:10.1103/physreva.44.2718

Cited by 92 publications

(52 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Because of the random presentation of the input vectors and (possibly) the random initialization of the weights, the learning process is a stochastic process governed by the master equation (Ritter and Schulten 1988;Heskes and Kappen 1991) ae(w', t)…”

Section: Transition Timesmentioning

confidence: 99%

Transition times in self-organizing maps

Heskes¹

1996

Biological Cybernetics

Self Cite

View full text Add to dashboard Cite

Abstract.We study the creation of topological maps. It is well known that topological defects, like kinks in one-dimensional maps or twists ('butterflies') in two-dimensional maps, can be (metastable) fixed points of the learning process. We are interested in transition times from these disordered configurations to the perfectly ordered configurations, i.e., the average time it takes to remove a kink or to unfold a twist. For this study we consider a self-organizing learning rule which is equivalent to the Kohonen learning rule, except for the determination of the 'winning' unit. The advantage of this particular learning rule is that it can be derived from an error potential. The existence of an error potential facilitates a global description of the learning process. Mappings in one and two dimensions are used as examples. For small lateral-interaction strength, topological defects correspond to local minima of the error potential, whereas global minima are perfectly ordered configurations. Theoretical results on the transition times from the local to the global minima of the error potential are compared with computer simulations of the learning rule.

show abstract

Section: Transition Timesmentioning

confidence: 99%

Transition times in self-organizing maps

Heskes¹

1996

Biological Cybernetics

Self Cite

View full text Add to dashboard Cite

show abstract

“…The learning parameter sets the typical scale of the weight change at each update. A large learning parameter leads to large fiuctuations in the network's representation [2].…”

Section: Introductionmentioning

confidence: 99%

Cooling schedules for learning in neural networks

1993

Self Cite

View full text Add to dashboard Cite

We derive cooling schedules for the global optimization of learning in neural networks. We discuss a two-level system with one global and one local minimum. The analysis is extended to systems with many minima. The optimal cooling schedule is (asymptotically) of the form g(t) =g*/lnt, with g(t) the learning parameter at time t and q* a constant, dependent on the reference learning parameters for the various transitions. In some simple cases, g can be calculated. Simulations confirm the theoretical results. PACS number(s): 87.10.+e

show abstract

“…INTRODUCTION Recently, some progress has been made in understanding learning processes in neural networks [1]. Instead of considering the dynamics of the fast variables, such as the spins in Hopfield-type neural networks, one focuses on the dynamics of the slow variables, the synapses and the thresholds.…”

mentioning

confidence: 99%

“…(1). Examples are backpropagation [2] for multilayered perceptrons, Kohonen-type learning [3,4] for topological maps, and Hebbian learning [5,6] [1] it was possible to compute these constants. In general, however, all the information needed to calculate these constants is not directly available.…”

mentioning

confidence: 99%

“…In the first place, the theory described in [1] (2). For an ensemble = of networks, the dynamics of the average network state can be found by expanding around the fixed point w" [1] g&w, )=-= « bw;)"&-=il« f (w, x))n&= (4) where the positive-definite matrix G is the first derivative of the average learning rule a& J;(w, x))" Equation (5) describes the dynamics of the average behavior of the stochastic process W(r). Our second assumption is that we can write this stochastic process as (7) With g white noise obeying (g)n=0 and (g )n=y, independent of w and q.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Learning-parameter adjustment in neural networks

Heskes

Kappen

1992

Phys. Rev. A

Self Cite

View full text Add to dashboard Cite

We present a learning-parameter adjustment algorithm, valid for a large class of learning rules in neural-network literature. The algorithm follows directly from a consideration of the statistics of the weights in the network. The characteristic behavior of the algorithm is calculated, both in a fixed and a changing environment. A simple example, Widrow-HofF learning for statistical classification, serves as an illustration. PACS number(s): 87.10.+e

show abstract

Learning processes in neural networks

Cited by 92 publications

References 36 publications

Transition times in self-organizing maps

Transition times in self-organizing maps

Cooling schedules for learning in neural networks

Learning-parameter adjustment in neural networks

Contact Info

Product

Resources

About