Sparse Neural Networks With Large Learning Diversity

Abstract

Coded recurrent neural networks with three levels of sparsity are introduced. The first level is related to the size of messages that are much smaller than the number of available neurons. The second one is provided by a particular coding rule, acting as a local constraint in the neural activity. The third one is a characteristic of the low final connection density of the network after the learning phase. Though the proposed network is very simple since it is based on binary neurons and binary connections, it is able to learn a large number of messages and recall them, even in presence of strong erasures. The performance of the network is assessed as a classifier and as an associative memory.

Nonbinary Associative Memory With Exponential Pattern Retrieval Capacity and Iterative Learning

“…Later, Gripon and Berrou came up with a different approach based on neural cliques, which increased the pattern retrieval capacity to O(n 2 ) [6]. Their method is based on dividing a neural network of size n into c clusters of size n/c each.…”

“…Next, we present a sufficient condition such that the minimum Hamming distance 6 between these exponential number of patterns is not too small. In order to prove such a result, we will exploit the expansion properties of the bipartite graph W ; our sufficient condition will be in terms of a lower bound on the parameters of the expander graph.…”

“…retrieve the correct memorized pattern from a given noisy version. Starting from the seminal work of Hopfield in 1982 [1], various artificial neural networks have been designed to mimic the task of the neuronal associative memory (see for instance [2], [3], [4], [5], [6]). …”

Sparse binary matrices as efficient associative memories

“…Recently, Gripon and Berrou [5], [6] introduced a novel architecture of associative memories based on the principles of modern error-correcting codes such as LDPC codes [7]. They proposed to store pieces of information in a multipartite binary graph by using cliques (a subset of fully interconnected nodes).…”

“…This is referred to as the "winner-takes-all" principle in the neuroscience literature. Gripon and Berrou [5], [6] claim to provide near-optimal efficiency, along with limited computational complexity and low error probability. The recent extension [8] proposes novel decoding rules that both enhance performance and yield convergence guarantees.…”