Effect of dilution in asymmetric recurrent neural networks

Abstract

We study with numerical simulation the possible limit behaviors of synchronous discrete-time deterministic recurrent neural networks composed of N binary neurons as a function of a network's level of dilution and asymmetry. The network dilution measures the fraction of neuron couples that are connected, and the network asymmetry measures to what extent the underlying connectivity matrix is asymmetric. For each given neural network, we study the dynamical evolution of all the different initial conditions, thus characterizing the full dynamical landscape without imposing any learning rule. Because of the deterministic dynamics, each trajectory converges to an attractor, that can be either a fixed point or a limit cycle. These attractors form the set of all the possible limit behaviors of the neural network. For each network we then determine the convergence times, the limit cycles' length, the number of attractors, and the sizes of the attractors' basin. We show that there are two network structures that maximize the number of possible limit behaviors. The first optimal network structure is fully-connected and symmetric. On the contrary, the second optimal network structure is highly sparse and asymmetric. The latter optimal is similar to what observed in different biological neuronal circuits. These observations lead us to hypothesize that independently from any given learning model, an efficient and effective biologic network that stores a number of limit behaviors close to its maximum capacity tends to develop a connectivity structure similar to one of the optimal networks we found.

Network dilution and asymmetry in an efficient brain

“…Similar values of the dilution are also found in monkey and mouse visual cortex [19,20], and in human cortex [21]: the reason for which evolution selected asymmetric and diluted connectivity for these areas is still today an open question, which has been so far nearly unexplored [22][23][24]. In our previous research [25], we found that a discrete-time RNN with McCulloch-Pitts neurons possess two network conditions with optimal memory storage, a completely connected and symmetric condition and a diluted asymmetric condition. Furthermore, we show that the optimal RNN condition of dilution and asymmetry corresponds to the dilution and asymmetry values found in the hippocampus and the neocortex.…”

“…This approach allows the direct comparison of the network properties given different symmetry ε and dilution ρ. Until [25], only completely connected McCulloch-Pitts discrete-time recurrent neural networks were considered, thus only networks that shared synaptic connection between all neurons were considered. Clearly, the assumption that any pair of neurons are connected by a synapse is not realistic if we consider brain regions with several neurons, because there would not be sufficient space for all the required connections.…”

“…The first is the Hebbian perspective in which memories are stored by rewiring the neural network until each neural pattern associated with a memory becomes a stable attractor [29]. The second is the innate framework in which memories are stored associating to each memory a pre-existing attractor [25] or several lego-like pre-existing attractors [30]. In both perspectives, given an input, a neural network evolves until it reaches the relative attractor, thus associating an input state (stimulus) to an output attractor (memory).…”

“…In both perspectives, given an input, a neural network evolves until it reaches the relative attractor, thus associating an input state (stimulus) to an output attractor (memory). Consequently, the number of attractors can be considered as an upper bound of the number of memories that can be stored [25]. Given this observation, it is reasonable to infer that neural networks with a single or few attractors may be considered to be inefficient memory devices with poor storing capability.…”

“…Meanwhile, the dilution expresses the fraction of edges J i,j that are missing, J i,j = 0. As in [25], to obtain a random network with a certain dilution value ρ and symmetry ε, we perform a convex sum over the matrix elements of a random symmetric matrix S ij , and a random anti-symmetric matrix A ij ,…”