Transient Chaotic Dimensionality Expansion by Recurrent Networks

Christian, Keup,; Kühn, Tobias; Dahmen, David; Helias, Moritz

doi:10.1103/physrevx.11.021064

Cited by 14 publications

(8 citation statements)

References 82 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Qualitative differences between continuous-time models and discrete-time models also have been suggested in Ref. [6,15].…”

Section: Effective Equation Of Motionmentioning

confidence: 91%

“…Owing to this simplicity, for these classes, we can analytically determine the phase diagram (introduced later in Sec. V A) in the parameter space [2,6,15,81] and evaluate various dynamical quantities under driving signals, such as the maximum Lyapunov exponent and the memory curve [15,60,77]. On the other hand, for more general universality classes such as the Gamma and the stable class, we need to deal with an infinite number of the self-consistent equations for all cumulants, which would be intractable without a numerical approach in general.…”

Section: Numerical Calculationmentioning

confidence: 99%

“…Artificial neural networks, originally designed to emulate biological network systems such as the human brain [1], serve as an essential basis of modern machine learning theory. These systems, especially recurrent random network models, exhibit rich dynamical properties, including collective chaotic dynamics [2][3][4][5][6], common signal-induced synchronization [7][8][9][10][11][12], and noise-induced suppression of chaos [13][14][15]. These dynamical aspects of random networks were investigated in great detail in early seminar works [2,3,13,16,17] and, more recently, formulated and improved elegantly in generating functional formalism [15,18].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Universality in reservoir computing and higher-order statistics

Haruna¹,

Toshio²,

Nakano³

2021

Preprint

View full text Add to dashboard Cite

In this work, we give a characterization of the reservoir computer (RC) by the network structure, especially the probability distribution of random coupling constants. First, based on the path integral method, we clarify the universal behavior of the random network dynamics in the thermodynamic limit, which depends only on the asymptotic behavior of the second cumulant generating functions of the network coupling constants. This result enables us to classify the random networks into several universality classes, according to the distribution function of coupling constants chosen for the networks. Interestingly, it is revealed that such a classification has a close relationship with the distribution of eigenvalues of the random coupling matrix. We also comment on the relation between our theory and a practical choice of random connectivity in the RC. Subsequently, we investigate the relationship between the RC's computational power and the network parameters for several universality classes. We perform several numerical simulations to evaluate the phase diagrams of the steady reservoir states, common signal-induced synchronization, and the computational power in the chaotic time-series predictions. As a result, we clarify the close relationship between these quantities, especially a remarkable computational performance near the phase transitions, which is realized near a non-chaotic transition boundary. These results may provide us with a new perspective on the designing principle for the RC.

show abstract

“…Qualitative differences between continuous-time models and discrete-time models also have been suggested in Ref. [6,15].…”

Section: Effective Equation Of Motionmentioning

confidence: 91%

Section: Numerical Calculationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Universality in reservoir computing and higher-order statistics

Haruna¹,

Toshio²,

Nakano³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Appropriate parameter selection of the model determines whether the algorithm can find the optimal solution quickly and accurately, which is always the difficulty faced by the TCNN class optimization model. Therefore, in order to facilitate rapid, effective and clear research and the use of appropriate parameter settings, this paper summarizes all parameter settings and selection guidance by referring to many literatures [1][2][3][4][5][6][7][8][9][10][17][18][19] and the above experimental analysis and verification, as shown in Table 2 To obtain good optimization performance of the model, it is necessary to properly select and balance the relationship between the basic parameters of the model and the MFCS parameter settings. The higher the complexity of the optimization problem, the stronger the non-monotony of MFCS is required.…”

Section: Mfcscnn Modelmentioning

confidence: 99%

“…By introducing the chaotic dynamic (self-feedback) term, TCNN can make use of the ergodic property, pseudo-randomness and non-repeatability of chaos to avoid local optimization [7,8]. However, TCNN's chaotic global search performance is still limited due to the parameter setting, excitation function, annealing function and other factors, resulting in not being ideal [9,10]. Therefore, scholars conduct comprehensive research from diverse perspectives to enhance the model's performance.…”

Section: Introductionmentioning

confidence: 99%

The Multiple Frequency Conversion Sinusoidal Chaotic Neural Network and Its Application

Hu,

Guo,

Wang

et al. 2023

Fractal Fract

View full text Add to dashboard Cite

Aiming at the problem that the global search performance of a transiently chaotic neural network is not ideal, a multiple frequency conversion sinusoidal chaotic neural network (MFCSCNN) model is proposed based on the biological mechanism of the brain, including multiple functional modules and sinusoidal signals of different frequencies. In this model, multiple FCS functions and Sigmoid functions with different phase angles were used to construct the excitation function of neurons in the form of weighted sum. In this paper, the inverted bifurcation diagram, Lyapunov exponential diagram and parameter range of the model are given. The dynamic characteristics of the model are analyzed and applied to function optimization and combinatorial optimization problems. Experimental results show that the multiple frequency conversion sinusoidal chaotic neural network has better global search performance than the transient chaotic neural network and other related models.

show abstract

Gradient-based learning drives robust representations in recurrent neural networks by balancing compression and expansion

et al. 2022

View full text Add to dashboard Cite

Recordings of neural circuits in the brain reveal extraordinary dynamical richness and high variability. At the same time, dimensionality reduction techniques generally uncover low-dimensional structures underlying these dynamics. What determines the dimensionality of activity in neural circuits? What is the functional role of this dimensionality in behavior and task learning? In this work we address these questions using recurrent neural network (RNN) models, which have recently shown promise in predicting and explaining brain dynamics. Through simulations and mathematical analysis, we show how the dimensionality of RNN activity evolves over time and over stages of learning. We find that RNNs can learn to balance tendencies to expand and compress dimensionality in a way that matches task demands and further generalizes to new data. Strongly chaotic networks appear particularly adept in learning this balance in the case of classifying low-dimensional inputs, combining the natural tendency of chaos to expand dimensionality with opportunistic compression driven by stochastic gradient descent to form representations with good generalization properties. These findings shed light on fundamental dynamical mechanisms by which neural networks solve tasks with robust representations that generalize to new cases.

show abstract

Transient Chaotic Dimensionality Expansion by Recurrent Networks

Cited by 14 publications

References 82 publications

Universality in reservoir computing and higher-order statistics

Universality in reservoir computing and higher-order statistics

The Multiple Frequency Conversion Sinusoidal Chaotic Neural Network and Its Application

Gradient-based learning drives robust representations in recurrent neural networks by balancing compression and expansion

Contact Info

Product

Resources

About