Optimal Sequence Memory in Driven Random Networks

Schuecker, Jannis; Goedeke, Sven; Helias, Moritz

doi:10.1103/physrevx.8.041029

Cited by 80 publications

(214 citation statements)

References 81 publications

Supporting

Mentioning

198

Contrasting

Order By: Relevance

“…An online supervised learning algorithm for RNNs proposed by Sussillo and Abbott [33] exhibits its best performance when their autonomous dynamics is adjusted to the chaotic region not far from the critical point where chaotic dynamics can be suppressed by input signals. Schuecker et al [23] showed that the network memory capacity for continuous-time nonlinear RNNs peaks in the ordered regime with g 2 > 1, which is consistent to our result. They argued that the dynamic suppression of chaos (DSC), which results from the fact that the onset of local instability precedes that of asymptotic instability, contributes to optimal information processing.…”

Section: Discussionsupporting

confidence: 92%

“…In ESNs, the shift of the critical g 2 * toward the chaotic regime induced by input signals is solely due to a mechanism called the static suppression of chaos (SSC), which increases the frequency with which an orbit visits the contracting region of the phase space. Unlike SSC, DSC is conjectured to occur based on fast switching among different unstable directions caused by input signals [23]. However, ESNs with leaky neurons [34] are expected to exhibit DSC [23].…”

Section: Discussionmentioning

confidence: 99%

“…The memory capacity of an ESN is defined as the quality of the optimal linear estimator of the past input s(t − n) using the present state of neurons x i (t). Following previous work [23,25], we assume a sparse readout, namely, there are K = O(1) readout neurons 1 ≤ i ≤ K and consider linear readoutŝ(t) = K i=1 v i x i (t). Given time-delay n (n = 1, 2, .…”

Section: B Memory Capacitymentioning

confidence: 99%

“…However, the memory capacity of nonlinear RNNs is difficult to study by analytical methods [22]. Recently, Schuecker et al [23] successfully derived an analytical expression for memory capacity for continuous-time nonlinear RNNs [24] in which each neuron is driven by independent input signals following a white-noise Gaussian process. Toyoizumi and Abbott [25] analytically calculated the signal-to-noise ratio, which is equivalent to the inverse of memory capacity at the limit of zero input strength, for discrete-time nonlinear RNNs driven by a common time-varying input signal.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Optimal short-term memory before the edge of chaos in driven random recurrent networks

Haruna

Nakajima

2019

Phys. Rev. E

View full text Add to dashboard Cite

The ability of discrete-time nonlinear recurrent neural networks to store time-varying small input signals is investigated by mean-field theory. The combination of a small input strength and mean-field assumptions makes it possible to derive an approximate expression for the conditional probability density of the state of a neuron given a past input signal. From this conditional probability density, we can analytically calculate short-term memory measures, such as memory capacity, mutual information, and Fisher information, and determine the relationships among these measures, which have not been clarified to date to the best of our knowledge. We show that the network contribution of these short-term memory measures peaks before the edge of chaos, where the dynamics of input-driven networks is stable but corresponding systems without input signals are unstable.

show abstract

Section: Discussionsupporting

confidence: 92%

Section: Discussionmentioning

confidence: 99%

Section: B Memory Capacitymentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Optimal short-term memory before the edge of chaos in driven random recurrent networks

Haruna

Nakajima

2019

Phys. Rev. E

View full text Add to dashboard Cite

show abstract

“…Other transitions obtained with this model regard the transient time spent to reach the attractor from random inputs [13][14][15] and the size of the basins of attraction of the existing limit cycles [14]. Adding noise [16,17], dilution [18], a gain function [19][20][21] or self-interaction [22] may further expand the array of possible transitions in the system. In particular, adding dilution to fully asymmetric networks results in an increased number of attractors, and therefore to an increased storage capacity [23].…”

Section: Introductionmentioning

confidence: 99%

On the number of limit cycles in asymmetric neural networks

et al. 2019

View full text Add to dashboard Cite

We consider the storage properties of temporal patterns, i.e. cycles of finite lengths, in neural networks represented by (generally asymmetric) spin glasses defined on random graphs. Inspired by the observation that dynamics on sparse systems has more basins of attractions than the dynamics of densely connected ones, we consider the attractors of a greedy dynamics in sparse topologies, considered as proxy for the stored memories. We enumerate them using numerical simulation and extend the analysis to large systems sizes using belief propagation. We find that the logarithm of the number of such cycles is a non monotonic function of the mean connectivity and we discuss the similarities with biological neural networks describing the memory capacity of the hippocampus. arXiv:2001.00262v1 [cond-mat.dis-nn] 1 Jan 2020

show abstract

Self‐Organization of Remote Reservoirs: Transferring Computation to Spatially Distant Locations

Tanaka

Tokudome

Minami

et al. 2021

Advanced Intelligent Systems

View full text Add to dashboard Cite

Soft materials generate rich and diverse dynamics that can be used as computational resources based on the framework of physical reservoir computing. Herein, a method that exploits the dynamic coupling between soft structures and a water medium to allow for the transfer of computation to spatially distant locations is proposed. This technique is implemented by introducing the concept of remote reservoirs that can autonomously alter their physical constituents in real time rather than using reservoirs with predefined, fixed physical constituents. These remote reservoirs self‐organize by including many ad hoc physical substrates in the environment, making the framework more flexible for soft robotic applications. Using a simple experimental platform consisting of silicone rubber strips containing embedded flexible strain sensors, it is demonstrated that the dynamics of passive silicone rubber strips located at a distance from the actuation point can be successfully exploited for computation through generalized synchronization realized via the medium of water. Future application scenarios are illustrated, in which the proposed technique is applied in emergency situations as a communication tool for transferring a message to a location that humans cannot easily access. For example, the technique could be applied to communicate with people trapped in submerged caves.

show abstract

Optimal Sequence Memory in Driven Random Networks

Cited by 80 publications

References 81 publications

Optimal short-term memory before the edge of chaos in driven random recurrent networks

Optimal short-term memory before the edge of chaos in driven random recurrent networks

On the number of limit cycles in asymmetric neural networks

Self‐Organization of Remote Reservoirs: Transferring Computation to Spatially Distant Locations

Contact Info

Product

Resources

About