Sparse binary matrices as efficient associative memories

Abstract

Associative memories are widely used devices which can be viewed as universal error-correcting decoders. Employing error-correcting code principles in these devices has allowed to greatly enhance their performance. In this paper we reintroduce a neural-based model using the formalism of linear algebra and extend its functionality, originally limited to erasure retrieval, to handle approximate inputs. In order to perform the retrieval, we use an iterative algorithm that provably converges. We then analyze the performance of the associative memory under the assumption of connection independence. We support our theoretical results with numerical simulations. I. INTRODUCTIONAssociative memories can serve as an alternative to indexed memories, where the content is retrieved from a part of it. The associative memories behave similarly to errorcorrecting decoders. They are used in many domains, including network routers [1], intrusion detection systems [2], database engines [3], and CPU caches [4].Recently, Gripon and Berrou [5], [6] introduced a novel architecture of associative memories based on the principles of modern error-correcting codes such as LDPC codes [7]. They proposed to store pieces of information in a multipartite binary graph by using cliques (a subset of fully interconnected nodes). The retrieving procedure instantiates a simple message-passing algorithm in two phases. First, the nodes sum their inputs. After that, an adaptive threshold is applied such that only those nodes which obtain the maximum score in their part remain activated. This is referred to as the "winner-takes-all" principle in the neuroscience literature. Gripon and Berrou [5], [6] claim to provide near-optimal efficiency, along with limited computational complexity and low error probability. The recent extension [8] proposes novel decoding rules that both enhance performance and yield convergence guarantees. However, the model of [8] has only been studied for the case of erasure channels so far, where input messages have some missing symbols. In order to adapt these devices to other application domains, such as machine learning and noise cancellation, it is of interest to study the ability of these memories to recover a message from an approximate input.While implementing these associative memories, it was shown in [9] that the original rule is a hard-thresholding instantiation, which uses a matrix product followed by a nonlinear thresholding. In [10], the authors propose to use only binary operations to perform the retrieval process, using a smart rewriting of what was originally proposed in [8].In this paper we introduce a linear-algebraic formalization of these associative memories and their functionality over