Investigating Echo-State Networks Dynamics by Means of Recurrence Analysis

Bianchi, Filippo Maria; Livi, Lorenzo; Alippi, Cesare

doi:10.1109/tnnls.2016.2630802

Cited by 84 publications

(58 citation statements)

References 47 publications

(68 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…8 Note that it holds N α,β ± α−1 α = − 1 β w ± (α). 9 Due to symmetry, this condition holds also when inverting the role of N d,c , N a,b and consequently (a, b), (d, c). Figure 13: In all panels, filled points denote stable attractors, while circles denote saddles or repellers.…”

Section: B2 Esn Maps In Two Dimensionsmentioning

confidence: 97%

See 1 more Smart Citation

Interpreting Recurrent Neural Networks Behaviour via Excitable Network Attractors

2019

Self Cite

View full text Add to dashboard Cite

Introduction: Machine learning provides fundamental tools both for scientific research and for the development of technologies with significant impact on society. It provides methods that facilitate the discovery of regularities in data and that give predictions without explicit knowledge of the rules governing a system. However, a price is paid for exploiting such flexibility: machine learning methods are typically black-boxes where it is difficult to fully understand what the machine is doing or how it is operating. This poses constraints on the applicability and explainability of such methods. Methods: Our research aims to open the black-box of recurrent neural networks, an important family of neural networks used for processing sequential data. We propose a novel methodology that provides a mechanistic interpretation of behaviour when solving a computational task. Our methodology uses mathematical constructs called excitable network attractors, which are invariant sets in phase space composed of stable attractors and excitable connections between them. Results and Discussion: As the behaviour of recurrent neural networks depends both on training and on inputs to the system, we introduce an algorithm to extract network attractors directly from the trajectory of a neural network while solving tasks. Simulations conducted on a controlled benchmark task confirm the relevance of these attractors for interpreting the behaviour of recurrent neural networks, at least for tasks that involve learning a finite number of stable states and transitions between them.as reservoir computing [32,33]. Echo state networks (ESNs) [21,30] constitute an important example of reservoir computing, where a recurrent layer (called a reservoir) is composed of a large number of neurons with randomly initialised connections that are not fine-tuned via gradient-based optimisation mechanisms. The main idea behind ESNs is to exploit the rich dynamics generated by the reservoir with an output layer, the read-out that is optimised to solve a specific task. Problem statement and research hypothesisThe high-dimensional and non-linear nature of RNNs complicates interpretability of their internal dynamics, which are characterised by complex, input-dependent spatio-temporal patterns of activity [47,55]. This poses constraints on understanding the behaviour of RNNs: they are usually viewed as black-boxes from which it is hard to extract useful knowledge about their inner workings. As highlighted by recent research efforts [10,24,40], similar interpretability issues affect many other machine learning methods. Furthermore, an increasing societal need to develop accountability and explainability of decision making by AI [17] is driving the development of methodologies for explaining the behaviour of such methods.Our aim in this paper is to develop effective models that capture the essential dynamical behaviour of RNNs on computational tasks as input-driven responses of a dynamical system, while neglecting microscopic details of the RNN dynamics in phase...

show abstract

Section: B2 Esn Maps In Two Dimensionsmentioning

confidence: 97%

“…Beyond the fold bifurcation, a new pair of fixed points is generated. there are exactly 5 fixed points 9.…”

mentioning

confidence: 99%

Interpreting Recurrent Neural Networks Behaviour via Excitable Network Attractors

2019

Self Cite

View full text Add to dashboard Cite

show abstract

“…Several authors have used hallmark indicators of criticality to contrive optimal reservoirs with enhanced performance. The measures employed include Lyapunov exponents [62,63,[98][99][100][101][102] and Fisher information [103,104] of reservoir dynamics. The former estimates how rapidly slight perturbations get amplified (thus diverge) as the reservoir dynamics unfold.…”

Section: B Good Reservoir Design: Separability Generalization Critmentioning

confidence: 99%

Evolutionary aspects of reservoir computing

Seoane

2019

Phil. Trans. R. Soc. B

View full text Add to dashboard Cite

Reservoir Computing (RC) is a powerful computational paradigm that allows high versatility with cheap learning. While other artificial intelligence approaches need exhaustive resources to specify their inner workings, RC is based on a reservoir with highly non-linear dynamics that does not require a fine tuning of its parts. These dynamics project input signals into high-dimensional spaces, where training linear readouts to extract input features is vastly simplified. Thus, inexpensive learning provides very powerful tools for decision making, controlling dynamical systems, classification, etc. RC also facilitates solving multiple tasks in parallel, resulting in a high throughput. Existing literature focuses on applications in artificial intelligence and neuroscience. We review this literature from an evolutionary perspective. RC's versatility make it a great candidate to solve outstanding problems in biology, which raises relevant questions: Is RC as abundant in Nature as its advantages should imply? Has it evolved? Once evolved, can it be easily sustained? Under what circumstances? (In other words, is RC an evolutionarily stable computing paradigm?) To tackle these issues we introduce a conceptual morphospace that would map computational selective pressures that could select for or against RC and other computing paradigms. This guides a speculative discussion about the questions above and allows us to propose a solid research line that brings together computation and evolution with RC as a working bench.

show abstract

“…In fact, the complexity of the network is designated by two parameters to be minimized which are the ARS and the ARCR. The only difference between both of scenarios is at the level of the fitness function which becomes: Minimize F(Particle)={Training RMSE, ARCR, ARS} where the ARCR, the ARS and the training RMSE are computed according to equations (10), (11) and (12).…”

Section:  Tri-objective Pso-esn-raementioning

confidence: 99%