Noise Facilitation in Associative Memories of Exponential Capacity

Abstract

Recent advances in associative memory design through structured pattern sets and graph-based inference algorithms have allowed reliable learning and recall of an exponential number of patterns that satisfy certain subspace constraints. Although these designs correct external errors in recall, they assume neurons that compute noiselessly, in contrast to the highly variable neurons in brain regions thought to operate associatively, such as hippocampus and olfactory cortex. Here we consider associative memories with boundedly noisy internal computations and analytically characterize performance. As long as the internal noise level is below a specified threshold, the error probability in the recall phase can be made exceedingly small. More surprising, we show that internal noise improves the performance of the recall phase while the pattern retrieval capacity remains intact: the number of stored patterns does not reduce with noise (up to a threshold). Computational experiments lend additional support to our theoretical analysis. This work suggests a functional benefit to noisy neurons in biological neuronal networks.

Associative Algorithms for Computational Creativity

“…Associative memories are used in a variety of information retrieval settings (Chen, ) and in computing applications including memory caches for processing units (Chisvin & Duckworth, ), relational databases (Lin, Smith, & Smith, ), among many others. Typical connectionist structures of associative memories include Hopfield neural networks (Hopfield, ), Kohonen maps (Kohonen, ), Boltzmann machines (Ackley, Hinton, & Sejnowski, ), and constructions based on sparse graph codes (Karbasi, Salavati, Shokrollahi, & Varshney, ); connectionist approaches have been described for creativity systems. Herein, however, we use symbolic algorithms for association.…”

Learning and Recovery in the ReLU Model

“…Apart from being closely related, one of our main motivations behind studying these two problems together comes from the recent work on associative memory Karbasi et al [2014], Rawat [2015, 2017]. An associative memory consists of a learning phase, where a generative model is learned from a given dataset; and a recovery phase, where given a noisy version of a data point generated by the generative model, the noise-free version is recovered with the help of the knowledge of the generative model.…”

Performance of LDPC Decoders With Missing Connections

“…Asymptotic characterizations were also determined for Gallager A [13] and Gallager B decoders with transient noise [14]- [16], energy optimization [17], and both permanent and transient noise [18]. Noisy decoding [19]- [25], and general noisy belief propagation, not necessarily in decoding [26], [27], have also been studied. Recent studies show bit-flipping decoders with data-dependent gate failures can achieve zero error probability [25], [28], but with a subset of computation hardware that is reliable and no wiring diagram errors.…”

“…SF in decoding was observed with transient errors in computation, rather than with missing connections, initially in memory recall[27],[29] and then in communications[23],[30].…”