Short-term Sequence Memory: Compressive effects of Recurrent Network Dynamics

Abstract

Neural networks have become a ubiquitous as cognitive models in neuroscience and as machine learning systems. Deep neural networks in particular are achieving near-human performance in many applications. More recently, recurrent neural networks (RNNs) are being increasingly utilized, both as standalone structures and as layers of deep networks. RNNs are especially interesting as cortical networks are recurrent, indicating that recurrent connections are important in human-level processing. Despite their growing use, theory on the computational properties of RNNs is sparse. As many applications hinge on RNNs accumulating information dynamically, the ability of RNNs to iteratively compress information into the network is particularly critical. We thus present here non-asymptotic bounds on the network's short-term memory (STM; the number of inputs that can be compressed into and recovered from a network state). Previous bounds on a random RNN's STM limit the number of recoverable inputs by the number of network nodes. We show that when the input sequences are sparse in a basis or the matrix inputs is low-rank, the number of network nodes needed to achieve an STM grows as the overall information rate. Thus RNNs can efficiently store information embedded in longer input streams, shedding light on their computational capabilities.

Multiplex visibility graphs to investigate recurrent neural network dynamics

“…Nowadays, RNNs are gaining renewed interest in neuroscience due to their biological plausibility78910 and in computer science and engineering for their modeling ability1112. RNNs are capable to generate complex dynamics and perform inference based on current inputs and internal state, the latter maintaining a vanishing memory of past inputs1314.…”