Efficient architectures for sparsely-connected high capacity associative memory models

Abstract

In physical implementations of associative memory, wiring costs play a significant role in shaping patterns of connectivity. In this study of sparsely-connected associative memory, a range of architectures is explored in search of optimal connection strategies which maximise pattern-completion performance, while at the same time minimising wiring costs. It is found that architectures in which the probability of connection between any two nodes is based on relatively tight Gaussian and exponential distributions perform well, and that for optimum performance, the width of the Gaussian distribution should be made proportional to the number of connections per node. It is also established from a study of other connection strategies that distal connections are not necessary for good pattern-completion performance. Convergence times and network scalability are also addressed in the wide ranging study.

Using a genetic algorithm to investigate efficient connectivity in associative memories

“…So the results suggest that large-scale computation can be accomplished in networks with predominantly local connectivity. Further work reported in (Calcraft, Adams et al 2007) confirms this suggestion with investigations of networks with many thousands of units. Moreover we have found that modular architectures that have similarity to cortical structures, such as mini-columns also produce parsimonious and well performing networks (Chen, Adams et al 2008) We are currently investigating the generality of our results and we are evaluating the importance of connectivity in networks of spiking neurons.…”

“…where µ is the maximum connection length and ν normalizes it. Both the exponential and the restricted linear distributions perform well (Calcraft, Adams et al 2007) -having a good balance of pattern completion ability and low mean wiring length.…”

Connection Strategies in Associative Memory Models with Spiking and Non-spiking Neurons

“…It has been established that there are indeed such patterns of connectivity; in particular several researchers have shown that so-called small world [6] connectivity can give good performance. We have also shown, that in non-spiking networks, fairly tight Gaussian distributions of connectivity can give very parsimonious networks [2]. In this paper we extend our analysis of how the connectivity affects performance to the more complex dynamics exhibited by networks of integrate and fire spiking neurons.…”

“…In earlier work [1][2][3] we have shown how the pattern of connectivity in sparsely connected, associative memories influences the functionality of the networks. The nodes in our networks are given a position, either in a 1D or 2D space.…”

Efficient connection strategies in 1D and 2D associative memory models with and without displaced connectivity

“…Firstly, the degree of clustering [4], the extent to which nodes connected to any given node are also connected to each other, decreases from 0.73 to 0.53 as we move from 1D to 2D in the above locallyconnected networks; and we have previously found that very tightly clustered networks perform badly as associators [16]. Secondly, there is an improvement in communication across the network as we increase dimensionality.…”