Phase diagram and storage capacity of sequence processing neural networks

Düring, Alexander; Coolen, Anthony C. C.; Sherrington, David C.

doi:10.1088/0305-4470/31/43/005

Cited by 47 publications

(88 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the case of L = 1 c = 1 L = 1.0 , which is fully connected with no delay elements, the recurrent neural network's storage capacity α C for sequential association is 0.269. This agrees with the results of the previous works [29,30,31]. As the length of delay L increases, storage capacity α C increases even though the total number of synapses is constant.…”

Section: Random Pruningsupporting

confidence: 92%

Storage Capacity Diverges With Synaptic Efficiency in an Associative Memory Model With Synaptic Delay and Pruning

Miyoshi

Okada

2004

IEEE Trans. Neural Netw.

View full text Add to dashboard Cite

It is known that storage capacity per synapse increases by synaptic pruning in the case of a correlation-type associative memory model. However, the storage capacity of the entire network then decreases. To overcome this difficulty, we propose decreasing the connecting rate while keeping the total number of synapses constant by introducing delayed synapses. In this paper, a discrete synchronous-type model with both delayed synapses and their prunings is discussed as a concrete example of the proposal. First, we explain the Yanai-Kim theory by employing the statistical neurodynamics. This theory involves macrodynamical equations for the dynamics of a network with serial delay elements. Next, considering the translational symmetry of the explained equations, we re-derive macroscopic steady state equations of the model by using the discrete Fourier transformation. The storage capacities are analyzed quantitatively. Furthermore, two types of synaptic prunings are treated analytically: random pruning and systematic pruning. As a result, it becomes clear that in both prunings, the storage capacity increases as the length of delay increases and the connecting rate of the synapses decreases when the total number of synapses is constant. Moreover, an interesting fact becomes clear: the storage capacity asymptotically approaches 2/π due to random pruning. In contrast, the storage capacity diverges in proportion to the logarithm of the length of delay by systematic pruning and the proportion constant is 4/π. These results theoretically support the significance of pruning following an overgrowth of synapses in the brain and strongly suggest that the brain prefers to store dynamic attractors such as sequences and limit cycles rather than equilibrium states.

show abstract

Section: Random Pruningsupporting

confidence: 92%

Storage Capacity Diverges With Synaptic Efficiency in an Associative Memory Model With Synaptic Delay and Pruning

Miyoshi

Okada

2004

IEEE Trans. Neural Netw.

View full text Add to dashboard Cite

show abstract

“…During et al [6] analyzed the properties of the stationary states (the storage capacity and the phase diagram) of a recurrent network for purely ASP, without pattern reconstruction, and a solution for the transient dynamics of the model was recently discussed [7]. The effects of stochastic noise on the model [4] were only recently analyzed in a feed-forward neural network architecture [1], in which an exact solution for the dynamics and complete phase diagrams of stationary states were obtained.…”

Section: Introductionmentioning

confidence: 99%

Cycles in symmetric sequence processing

Metz¹,

Theumann²

2007

AIP Conference Proceedings

View full text Add to dashboard Cite

Abstract. The competition between pattern reconstruction and sequence processing is studied here in an exactly solvable feed-forward layered neural network model of binary units and patterns near saturation. We show results for both symmetric and asymmetric sequence processing, either one competing with pattern reconstruction represented by a Hebbian interaction, in order to compare these two kinds of sequence processing. Phase diagrams of stationary states are obtained and a new phase of cycles of period two is found for a weak Hebbian term in the case of symmetric sequence processing, independently of the number of condensed patterns c which have macroscopic overlaps with the states of the network. In contrast, the stability of these cycles depends strongly on c. These results are in contrast with those for the competition between a Hebbian interaction and an asymmetric sequence processing interaction [1], in which the period of the cycles is c and the stability of these solutions does not depend on c. The dynamics of the macroscopic overlaps in the stationary cyclic phase is analyzed in both models.

show abstract

“…Also for networks that act as memory for sequences, capacities have been calculated in both the perceptron (Nadal, 1991) and the attractor case (Herz, Li, & van Hemmen, 1991). An important result is that the capacity of sequence memory in Hopfield-type networks is about twice as large as that of a static attractor network (Düring, Coolen, & Sherington, 1998).…”

Section: Introductionmentioning

confidence: 99%