Dynamical Learning of Dynamics

Klos, Christian; Kossio, Yaroslav Felipe Kalle; Goedeke, Sven; Gilra, Aditya; Memmesheimer, Raoul-Martin

doi:10.1103/physrevlett.125.088103

Cited by 44 publications

(22 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…bounded activity). Our work may also bring a novel perspective in training recurrent networks with feedback in the line of the FORCE algorithm, where the focus is on generating patterns (time series) that consist of trajectories (Sussillo and Abbott, 2009;Miconi, 2017;Klos et al, 2020). A possible extension is to generate time series with desired covariance-based patterns instead.…”

Section: Discussionmentioning

confidence: 99%

Covariance-based information processing in reservoir computing systems

Lawrie¹,

Moreno-Bote

Gilson

2021

Preprint

View full text Add to dashboard Cite

In biological neuronal networks, information representation and processing are achieved through plasticity learning rules that have been empirically characterized as sensitive to second and higher-order statistics in spike trains. However, most models in both computational neuroscience and machine learning aim to convert diverse statistical properties in inputs into first-order statistics in outputs, like in modern deep learning tools. In the context of classification, such schemes have merit for inputs like static images, but they are not well suited to capture the temporal structure in time series. In contrast, the recently developed covariance perceptron uses second-order statistics by mapping input covariances to output covariances in a consistent fashion. Here, we explore the applicability of covariance-based perceptron readouts in reservoir computing networks to classify synthetic multivariate time series structured at different statistical orders (first and second). We show that the second-order framework outperforms or matches the classical mean paradigm in terms of accuracy. We expose nontrivial relationships between input, reservoir and output dynamics, which suggest an important role for recurrent connectivity in transforming information representations in biologically inspired architectures. Finally, we solve a real automatic speech recognition task for the classification of spoken digits to further demonstrate the potential of covariance-based decoding.

show abstract

Section: Discussionmentioning

confidence: 99%

Covariance-based information processing in reservoir computing systems

Lawrie¹,

Moreno-Bote

Gilson

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Our results therefore uncovered a novel computational role of contextual inputs for generalization. This generalization relied on continuous, parametric control by the contextual cue, a mechanism related to recent reports of parametric control of non-linear dynamics in RNNs (Klos et al, 2020;Kim et al, 2021). These studies did not focus on the role of dimensionality, but interestingly relied on specific training schemes that implicitly induce a low-rank structure (Sussillo and Abbott, 2009).…”

Section: Discussionmentioning

confidence: 99%

Parametric control of flexible timing through low-dimensional neural manifolds

Beirán

Meirhaeghe

Sohn

et al. 2021

Preprint

View full text Add to dashboard Cite

Biological brains possess an unparalleled ability to generalize adaptive behavioral responses from only a few examples. How neural processes enable this capacity to extrapolate is a fundamental open question. A prominent but underexplored hypothesis suggests that generalization is facilitated by a low-dimensional organization of collective neural activity. Here we tested this hypothesis in the framework of flexible timing tasks where dynamics play a key role. Examining trained recurrent neural networks we found that confining the dynamics to a low-dimensional subspace allowed tonic inputs to parametrically control the overall input-output transform and enabled smooth extrapolation to inputs well beyond the training range. Reverse-engineering and theoretical analyses demonstrated that this parametric control of extrapolation relies on a mechanism where tonic inputs modulate the dynamics along non-linear manifolds in activity space while preserving their geometry. Comparisons with neural data from behaving monkeys confirmed the geometric and dynamical signatures of this mechanism.

show abstract

“…By weighting the reservoir states, the output layer computes the prediction [6][7][8]. Standard echo state networks efficiently predict chaotic time series of ordinary systems with high accuracy [9][10][11][12][13][14][15]. Furthermore, hybrid models based on the combination of model-free reservoir computing with a mathematical model of the task already showed improved performance in chaotic time series prediction.…”

mentioning

confidence: 99%

Learn one size to infer all: Exploiting translational symmetries in delay-dynamical and spatio-temporal systems using scalable neural networks

Goldmann¹,

Mirasso²,

Fischer³

et al. 2021

Preprint

View full text Add to dashboard Cite

Caused by finite signal propagation velocities, many complex systems feature time delays that may induce high-dimensional chaotic behavior and make forecasting intricate. Here, we propose an echo state network adaptable to the physics of systems with arbitrary delays. After training the network to forecast a system with a unique and sufficiently long delay, it already learned to predict the system dynamics for all other delays. A simple adaptation of the network's topology allows us to infer untrained features such as high-dimensional chaotic attractors, bifurcations, and even multistabilities, that emerge with shorter and longer delays. Thus, the fusion of physical knowledge of the delay system and data-driven machine learning yields a model with high generalization capabilities and unprecedented prediction accuracy.

show abstract

Dynamical Learning of Dynamics

Cited by 44 publications

References 47 publications

Covariance-based information processing in reservoir computing systems

Covariance-based information processing in reservoir computing systems

Parametric control of flexible timing through low-dimensional neural manifolds

Learn one size to infer all: Exploiting translational symmetries in delay-dynamical and spatio-temporal systems using scalable neural networks

Contact Info

Product

Resources

About