Design Strategies for Weight Matrices of Echo State Networks

Strauß, Tobias; Wustlich, Welf; Labahn, Roger

doi:10.1162/neco_a_00374

Cited by 77 publications

(49 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Interestingly, the simple SOGP with the isotropic Gaussian kernel also performed comparably well on the MackeyGlass, Ikeda, NP2 and NARMA-10 benchmarks, echoing recent results [23], [41] that complex reservoirs are not necessary for certain prediction tasks. We had expected the SOGP with the full ARD kernel to realise lower errors than the isotropic kernel because the additional lengthscale parameters would allow a better fit to the given problems.…”

Section: B Accuracy Resultssupporting

confidence: 70%

Spatio-Temporal Learning With the Online Finite and Infinite Echo-State Gaussian Processes

Soh

Demiris

2015

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

Successful biological systems adapt to change. In this work, we are principally concerned with adaptive systems that operate in environments where data arrives sequentially and is multi-variate in nature, e.g., sensory streams in robotic systems. We contribute two reservoir inspired methods: (1) the online echo-state Gaussian process (OESGP) and (2) its infinite variant, the online infinite echo-state Gaussian process (OIESGP). Both algorithms are iterative fixed-budget methods that learn from noisy time-series. In particular, the OESGP combines the echo-state network (ESN) with Bayesian online learning for Gaussian processes (GPs). Extending this to infinite reservoirs yields the OIESGP, which uses a novel recursive kernel with automatic relevance determination (ARD) that enables spatial and temporal feature weighting. When fused with stochastic natural gradient descent (SNGD), the kernel hyperparameters are iteratively adapted to better model the target system. Furthermore, insights into the underlying system can be gleamed from inspection of the resulting hyperparameters. Experiments on noisy benchmark problems (one-step prediction and system identification) demonstrate that our methods yields high accuracies relative to state-of-the-art methods and standard kernels with sliding windows, particularly on problems with irrelevant dimensions. In addition, we describe two case-studies in robotic learning-by-demonstration (LbD) involving the Nao humanoid robot and the ARTY smart wheelchair.

show abstract

Section: B Accuracy Resultssupporting

confidence: 70%

Spatio-Temporal Learning With the Online Finite and Infinite Echo-State Gaussian Processes

Soh

Demiris

2015

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

show abstract

“…A relevant class of reservoir variants in this regard is given by ESNs with orthogonal recurrent matrices [29,14], which were shown to lead to improved performance with respect to random reservoirs both in terms of memorization skills and in terms of predictive performance on non-linear tasks. In particular, reservoirs whose structure is based on permutation matrices represent particularly appealing instances of orthogonal ESNs [14,25], entailing a simple and very sparse pattern of connectivity among the recurrent units. Other relevant architectural variants are given by reservoirs structured according to a ring topology or to form a chain of units [23,25].…”

Section: Introductionmentioning

confidence: 99%

Reservoir Topology in Deep Echo State Networks

Gallicchio

Micheli

2019

Artificial Neural Networks and Machine Learning – ICANN 2019: Workshop and Special Sessions

View full text Add to dashboard Cite

Deep Echo State Networks (DeepESNs) recently extended the applicability of Reservoir Computing (RC) methods towards the field of deep learning. In this paper we study the impact of constrained reservoir topologies in the architectural design of deep reservoirs, through numerical experiments on several RC benchmarks. The major outcome of our investigation is to show the remarkable effect, in terms of predictive performance gain, achieved by the synergy between a deep reservoir construction and a structured organization of the recurrent units in each layer. Our results also indicate that a particularly advantageous architectural setting is obtained in correspondence of DeepESNs where reservoir units are structured according to a permutation recurrent matrix.

show abstract

“…Chaos is characterized by trajectories diverging exponentially fast. This can be quantified with L max (10) and DIV (11), whose values would be, respectively, very low and close to one for systems with a high degree of chaoticity. As an example, we consider a chaotic system obtained by LM configured with τ LM = 4.…”

Section: Visualization and Classification Of Reservoir Dynamicsmentioning

confidence: 99%

“…A popular reservoir-computing architecture is the echo-state network (ESN) [10], an RNN with a nontrainable, sparse recurrent reservoir, and an adaptable (usually) linear readout, mapping the reservoir to the output. ESN reservoir characterization and design attracted significant research efforts in the last decade [11]- [13]. This is mostly due to the puzzling behavior of the reservoir, which, although randomly initialized, has shown to be effective in modeling nonlinear dynamical systems of various nature [14]- [18].…”

mentioning

confidence: 99%

Investigating Echo-State Networks Dynamics by Means of Recurrence Analysis

Bianchi

Livi

Alippi

2018

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

Abstract-In this paper, we elaborate over the well-known interpretability issue in echo-state networks (ESNs). The idea is to investigate the dynamics of reservoir neurons with timeseries analysis techniques developed in complex systems research. Notably, we analyze time series of neuron activations with recurrence plots (RPs) and recurrence quantification analysis (RQA), which permit to visualize and characterize highdimensional dynamical systems. We show that this approach is useful in a number of ways. First, the 2-D representation offered by RPs provides a visualization of the high-dimensional reservoir dynamics. Our results suggest that, if the network is stable, reservoir and input generate similar line patterns in the respective RPs. Conversely, as the ESN becomes unstable, the patterns in the RP of the reservoir change. As a second result, we show that an RQA measure, called L max , is highly correlated with the well-established maximal local Lyapunov exponent. This suggests that complexity measures based on RP diagonal lines distribution can quantify network stability. Finally, our analysis shows that all RQA measures fluctuate on the proximity of the so-called edge of stability, where an ESN typically achieves maximum computational capability. We leverage on this property to determine the edge of stability and show that our criterion is more accurate than two well-known counterparts, both based on the Jacobian matrix of the reservoir. Therefore, we claim that RPs and RQA-based analyses are valuable tools to design an ESN, given a specific problem.Index Terms-Dynamics, echo-state network (ESN), recurrence plot (RP), recurrence quantification analysis (RQA), nonlinear time series analysis.

show abstract

Design Strategies for Weight Matrices of Echo State Networks

Cited by 77 publications

References 4 publications

Spatio-Temporal Learning With the Online Finite and Infinite Echo-State Gaussian Processes

Spatio-Temporal Learning With the Online Finite and Infinite Echo-State Gaussian Processes

Reservoir Topology in Deep Echo State Networks

Investigating Echo-State Networks Dynamics by Means of Recurrence Analysis

Contact Info

Product

Resources

About