Optimal short-term memory before the edge of chaos in driven random recurrent networks

Haruna, Taichi; Nakajima, Kohei

doi:10.1103/physreve.100.062312

Cited by 21 publications

(13 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the time-series prediction task, we observe a remarkable improvement of the RC's performance near the phase transition of its attractor. Although our result for the Gaussian network is essentially equivalent to the one discussed in previous studies [54][55][56][57][58][59][60], it provides us with a new perspective on the boundary between chaotic and polarized ordered phases. Particularly interesting is that the Gamma network, where higher-order statistics play a crucial role, exhibits computational improvement even near the boundary of a non-chaotic phase.…”

Section: Introductionsupporting

confidence: 71%

“…Especially for the case where J ij is a Gaussian variate, the mean-field theory for these discrete-time models has already been investigated in several kinds of literature [4,13,60,77]. Qualitative differences between continuous-time models and discrete-time models also have been suggested in Ref.…”

Section: Effective Equation Of Motionmentioning

confidence: 99%

“…Recurrent random network models exhibit various dynamical phases, such as chaotic or ordered phases, depending on the value of the parameters. Although some controversies [53] exist, RCs are believed to show long temporal memory or the best computational performance near the boundary between a stable and an unstable dynamics regime, that is, the edge of chaos [54][55][56][57][58][59][60]. Additionally, it has also been shown that decoding signals from these networks are robust to noise above and near the transition to chaos [4].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Universality in reservoir computing and higher-order statistics

Haruna¹,

Toshio²,

Nakano³

2021

Preprint

View full text Add to dashboard Cite

In this work, we give a characterization of the reservoir computer (RC) by the network structure, especially the probability distribution of random coupling constants. First, based on the path integral method, we clarify the universal behavior of the random network dynamics in the thermodynamic limit, which depends only on the asymptotic behavior of the second cumulant generating functions of the network coupling constants. This result enables us to classify the random networks into several universality classes, according to the distribution function of coupling constants chosen for the networks. Interestingly, it is revealed that such a classification has a close relationship with the distribution of eigenvalues of the random coupling matrix. We also comment on the relation between our theory and a practical choice of random connectivity in the RC. Subsequently, we investigate the relationship between the RC's computational power and the network parameters for several universality classes. We perform several numerical simulations to evaluate the phase diagrams of the steady reservoir states, common signal-induced synchronization, and the computational power in the chaotic time-series predictions. As a result, we clarify the close relationship between these quantities, especially a remarkable computational performance near the phase transitions, which is realized near a non-chaotic transition boundary. These results may provide us with a new perspective on the designing principle for the RC.

show abstract

Section: Introductionsupporting

confidence: 71%

Section: Effective Equation Of Motionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Universality in reservoir computing and higher-order statistics

Haruna¹,

Toshio²,

Nakano³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The typical case is evaluating how well the given reservoir can output the previous input sequence, and this measure is called memory capacity [46]. Focusing on ESN, the behaviors of memory capacity and their related measures are studied in detail with a linear activation function [46][47][48][49][50][51] and, recently, with a nonlinear activation function [52][53][54]. This measure is further generalized and extended to be able to evaluate the nonlinear memory capacities by decomposing the function φ into the combinations of multiple orthogonal polynomials [55], and the trade-off between the expressiveness of φ for linear and nonlinear functions is investigated [55,56].…”

Section: Prerequisite For a Successful Reservoirmentioning

confidence: 99%

Physical reservoir computing—an introductory perspective

Nakajima

2020

Jpn. J. Appl. Phys.

Self Cite

284

174

View full text Add to dashboard Cite

Understanding the fundamental relationships between physics and its information-processing capability has been an active research topic for many years. Physical reservoir computing is a recently introduced framework that allows one to exploit the complex dynamics of physical systems as information-processing devices. This framework is particularly suited for edge computing devices, in which information processing is incorporated at the edge (e.g., into sensors) in a decentralized manner to reduce the adaptation delay caused by data transmission overhead. This paper aims to illustrate the potentials of the framework using examples from soft robotics and to provide a concise overview focusing on the basic motivations for introducing it, which stem from a number of fields, including machine learning, nonlinear dynamical systems, biological science, materials science, and physics.

show abstract

“…Still, another domain of complexity-chaos-has been far less understood [16,17]. Even though enormous theory/data-driven studies on forecasting chaotic time series by means of recurrent NN have been conducted [18][19][20][21][22][23][24][25], there is still a lack of consensus on which features play the most important roles in the forecasting methods.…”

Section: Introductionmentioning

confidence: 99%

Entanglement-Structured LSTM Boosts Chaotic Time Series Forecasting

Meng

Yang

2021

Entropy

View full text Add to dashboard Cite

Traditional machine-learning methods are inefficient in capturing chaos in nonlinear dynamical systems, especially when the time difference Δt between consecutive steps is so large that the extracted time series looks apparently random. Here, we introduce a new long-short-term-memory (LSTM)-based recurrent architecture by tensorizing the cell-state-to-state propagation therein, maintaining the long-term memory feature of LSTM, while simultaneously enhancing the learning of short-term nonlinear complexity. We stress that the global minima of training can be most efficiently reached by our tensor structure where all nonlinear terms, up to some polynomial order, are treated explicitly and weighted equally. The efficiency and generality of our architecture are systematically investigated and tested through theoretical analysis and experimental examinations. In our design, we have explicitly used two different many-body entanglement structures—matrix product states (MPS) and the multiscale entanglement renormalization ansatz (MERA)—as physics-inspired tensor decomposition techniques, from which we find that MERA generally performs better than MPS, hence conjecturing that the learnability of chaos is determined not only by the number of free parameters but also the tensor complexity—recognized as how entanglement entropy scales with varying matricization of the tensor.

show abstract

Optimal short-term memory before the edge of chaos in driven random recurrent networks

Cited by 21 publications

References 34 publications

Universality in reservoir computing and higher-order statistics

Universality in reservoir computing and higher-order statistics

Physical reservoir computing—an introductory perspective

Entanglement-Structured LSTM Boosts Chaotic Time Series Forecasting

Contact Info

Product

Resources

About