Computational analysis of memory capacity in echo state networks

Farkaš, Igor; Bosák, Radomír; Gergeź, Peter

doi:10.1016/j.neunet.2016.07.012

Cited by 66 publications

(46 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hence, it seems reasonable for a dynamical system to be near the critical point for optimal memory capacity. However, it has also been pointed out that the dependence on network pa- * tharuna@lab.twcu.ac.jp † k nakajima@mech.t.u-tokyo.ac.jp rameters is not straightforward based on a systematic numerical simulation [17]. For linear RNNs, detailed analytic studies of memory capacity can be performed for both discrete-time [18,19] and continuous-time systems [20].…”

Section: Introductionmentioning

confidence: 99%

Optimal short-term memory before the edge of chaos in driven random recurrent networks

Haruna

Nakajima

2019

Phys. Rev. E

View full text Add to dashboard Cite

The ability of discrete-time nonlinear recurrent neural networks to store time-varying small input signals is investigated by mean-field theory. The combination of a small input strength and mean-field assumptions makes it possible to derive an approximate expression for the conditional probability density of the state of a neuron given a past input signal. From this conditional probability density, we can analytically calculate short-term memory measures, such as memory capacity, mutual information, and Fisher information, and determine the relationships among these measures, which have not been clarified to date to the best of our knowledge. We show that the network contribution of these short-term memory measures peaks before the edge of chaos, where the dynamics of input-driven networks is stable but corresponding systems without input signals are unstable.

show abstract

Section: Introductionmentioning

confidence: 99%

Optimal short-term memory before the edge of chaos in driven random recurrent networks

Haruna

Nakajima

2019

Phys. Rev. E

View full text Add to dashboard Cite

show abstract

“…This leads to a rather practical question: how much of the theory surrounding optimal reservoirs, based on maximizing memory capacity [5][6][7][8][9], is misleading if the ultimate goal is to maximize predictive power?…”

mentioning

confidence: 99%

Difference between memory and prediction in linear recurrent networks

Marzen

2017

Phys. Rev. E

View full text Add to dashboard Cite

Recurrent networks are trained to memorize their input better, often in the hopes that such training will increase the ability of the network to predict. We show that networks designed to memorize input can be arbitrarily bad at prediction. We also find, for several types of inputs, that one-node networks optimized for prediction are nearly at upper bounds on predictive capacity given by Wiener filters, and are roughly equivalent in performance to randomly generated five-node networks. Our results suggest that maximizing memory capacity leads to very different networks than maximizing predictive capacity, and that optimizing recurrent weights can decrease reservoir size by half an order of magnitude. Often, we remember for the sake of prediction. Such is the case, it seems, in the field of echo state networks (ESNs) [1,2]. ESNs are large input-dependent recurrent networks in which a "readout layer" is trained to match a desired output signal from the present network state. Sometimes, the desired output signal is the past or future of the input to the network.If the recurrent networks are large enough, they should have enough information about the past of the input signal to reproduce a past input or predict a future input well, and only the readout layer need be trained. Still, the weights and structure of the recurrent network can greatly affect the predictive capabilities of the recurrent network, and so many researchers are now interested in optimizing the network itself to maximize task performance [3].Much of the theory surrounding echo state networks centers on memorizing white noise, an input for which memory is essentially useless for prediction [4]. This leads to a rather practical question: how much of the theory surrounding optimal reservoirs, based on maximizing memory capacity [5][6][7][8][9], is misleading if the ultimate goal is to maximize predictive power?We study the difference between optimizing for memory and optimizing for prediction in linear recurrent networks subject to scalar temporally-correlated input generated by countable Hidden Markov models. Ref.[10] gave closed-form expressions for memory function of continuous-time linear recurrent networks in terms of the autocorrelation function of the input, and closely studied the case of an exponential autocorrelation function. * semarzen@mit.edu Ref.[11] gave similar expressions for discrete-time linear recurrent networks. Ref.[12] gave closed-form expressions for the Fisher memory curve of discrete-time linear recurrent networks, which measure how much changes in input signal perturb the network state; for linear recurrent networks, this curve is independent of the particular input signal.We differ from these previous efforts mostly in that we study both memory capacity and newly-defined "predictive capacity". We derive an upper bound for predictive capacity via Wiener filters in terms of the autocorrelation function of the input. Two surprising findings result. First, predictive capacity is not typically maximized at the "edge of critical...

show abstract

“…If, however, R is significantly smaller than 1, information is lost too fast over time, which is detrimental for tasks involving sequential memory. A spectral radius of about one is hence best [6], in the sense that it provides a maximal memory capacity if the network operates in a linear regime [7,8]. Similarly, Boedecker et al found the best memory capacity to be close to the largest Lyapunov exponent being equal to zero [9], which can be understood as a generalization of the aforementioned results for nonlinear dynamics.…”

Section: Introductionmentioning

confidence: 95%

Local homeostatic regulation of the spectral radius of echo-state networks

Schubert

Gros

2020

Preprint

View full text Add to dashboard Cite

Recurrent cortical network dynamics plays a crucial role for sequential information processing in the brain. While the theoretical framework of reservoir computing provides a conceptual basis for the understanding of recurrent neural computation, it often requires manual adjustments of global network parameters, in particular of the spectral radius of the recurrent synaptic weight matrix. Being a mathematical and relatively complex quantity, the spectral radius is not readily accessible to biological neural networks, which generally adhere to the principle that information about the network state should either be encoded in local intrinsic dynamical quantities (e.g. membrane potentials), or transmitted via synaptic connectivity. We present two synaptic scaling rules for echo state networks that solely rely on locally accessible variables. Both rules work online, in the presence of a continuous stream of input signals. The first rule, termed flow control, is based on a local comparison between the mean squared recurrent membrane potential and the mean squared activity of the neuron itself. It is derived from a global scaling condition on the dynamic flow of neural activities and requires the separability of external and recurrent input currents. We gained further insight into the adaptation dynamics of flow control by using a mean field approximation on the variances of neural activities that allowed us to describe the interplay between network activity and adaptation as a two-dimensional dynamical system. The second rule that we considered, variance control, directly regulates the variance of neural activities by locally scaling the recurrent synaptic weights. The target set point of this homeostatic mechanism is dynamically determined as a function of the variance of the locally measured external input. This functional relation was derived from the same mean-field approach that was used to describe the approximate dynamics of flow control.The effectiveness of the presented mechanisms was tested numerically using different external input protocols. The network performance after adaptation was evaluated by training the network to perform a time delayed XOR operation on binary sequences. As our main result, we found that flow control can reliably regulate the spectral radius under different input statistics, but precise tuning is negatively affected by interneural correlations. Furthermore, flow control showed a consistent task performance over a wide range of input strengths/variances. Variance control, on the other side, did not yield the desired spectral radii with the same precision. Moreover, task performance was less consistent across different input strengths.Given the better performance and simpler mathematical form of flow control, we concluded that a local control of the spectral radius via an implicit adaptation scheme is a realistic alternative to approaches using classical “set point” homeostatic feedback controls of neural firing.Author summaryHow can a neural network control its recurrent synaptic strengths such that network dynamics are optimal for sequential information processing? An important quantity in this respect, the spectral radius of the recurrent synaptic weight matrix, is a non-local quantity. Therefore, a direct calculation of the spectral radius is not feasible for biological networks. However, we show that there exist a local and biologically plausible adaptation mechanism, flow control, which allows to control the recurrent weight spectral radius while the network is operating under the influence of external inputs. Flow control is based on a theorem of random matrix theory, which is applicable if inter-synaptic correlations are weak. We apply the new adaption rule to echo-state networks having the task to perform a time-delayed XOR operation on random binary input sequences. We find that flow-controlled networks can adapt to a wide range of input strengths while retaining essentially constant task performance.

show abstract

Computational analysis of memory capacity in echo state networks

Cited by 66 publications

References 23 publications

Optimal short-term memory before the edge of chaos in driven random recurrent networks

Optimal short-term memory before the edge of chaos in driven random recurrent networks

Difference between memory and prediction in linear recurrent networks

Local homeostatic regulation of the spectral radius of echo-state networks

Contact Info

Product

Resources

About