Deep reservoir computing: A critical experimental analysis

Gallicchio, Claudio; Micheli, Alessio; Pedrelli, Luca

doi:10.1016/j.neucom.2016.12.089

Cited by 373 publications

(289 citation statements)

References 11 publications

Supporting

Mentioning

288

Contrasting

Order By: Relevance

“…Here, tanh indicates the element-wise application of the hyperbolic tangent nonlinearity, u(t) ∈ R NU represents the external input at time-step t, while W in , W (l) andŴ (l) respectively denote the input weight matrix (that modulates the external input stimulation to the first layer), the inter-layer weight matrix for layer l (that modulates the strength of the connections from layer l − 1 to layer l), and the recurrent reservoir weight matrix for layer l. In both the above equations 1 and 2 we omitted the bias terms for the ease of notation. The interested reader can find in [7] a more detailed description of the deep reservoir equations, framed in the more general context of leaky integrator reservoir units. In order to set up initial conditions for the state update equations 1 and 2, at time-step 0 all reservoir layers are set to a null state, i.e.…”

Section: Deep Echo State Networkmentioning

confidence: 99%

Reservoir Topology in Deep Echo State Networks

Gallicchio

Micheli

2019

Artificial Neural Networks and Machine Learning – ICANN 2019: Workshop and Special Sessions

Self Cite

View full text Add to dashboard Cite

Deep Echo State Networks (DeepESNs) recently extended the applicability of Reservoir Computing (RC) methods towards the field of deep learning. In this paper we study the impact of constrained reservoir topologies in the architectural design of deep reservoirs, through numerical experiments on several RC benchmarks. The major outcome of our investigation is to show the remarkable effect, in terms of predictive performance gain, achieved by the synergy between a deep reservoir construction and a structured organization of the recurrent units in each layer. Our results also indicate that a particularly advantageous architectural setting is obtained in correspondence of DeepESNs where reservoir units are structured according to a permutation recurrent matrix.

show abstract

Section: Deep Echo State Networkmentioning

confidence: 99%

Reservoir Topology in Deep Echo State Networks

Gallicchio

Micheli

2019

Artificial Neural Networks and Machine Learning – ICANN 2019: Workshop and Special Sessions

Self Cite

View full text Add to dashboard Cite

show abstract

“…The above analysis also shows that the time depends on the training process, which is closely relevant to the longterm time series prediction 4 . Next, we compare the performance of the given method with that of two benchmark methods (LSTNet [55] and DeepESN [51]) to check if they can improve predictive performance, especially for longterm series prediction. Tab.…”

Section: The Evolution Of Deep Echo State Network For Time Series Prmentioning

confidence: 99%

The expressivity and training of deep neural networks: Toward the edge of chaos?

Zhang

Li²,

Shen

et al. 2020

Neurocomputing

View full text Add to dashboard Cite

Expressivity is one of the most significant issues in assessing neural networks. In this paper, we provide a quantitative analysis of the expressivity for the deep neural network (DNN) from its dynamic model, where the Hilbert space is employed to analyze the convergence and criticality. We study the feature mapping of several widely used activation functions obtained by Hermite polynomials, and find sharp declines or even saddle points in the feature space, which stagnate the information transfer in DNNs. We then present a new activation function design based on the Hermite polynomials for better utilization of spatial representation. Moreover, we analyze the information transfer of DNNs, emphasizing the convergence problem caused by the mismatch between input and topological structure. We also study the effects of input perturbations and regularization operators on critical expressivity. Our theoretical analysis reveals that DNNs use spatial domains for information representation and evolve to the edge of chaos as depth increases. In actual training, whether a particular network can ultimately arrive the edge of chaos depends on its ability to overcome convergence and pass information to the required network depth. Finally, we demonstrate the empirical performance of the proposed hypothesis via multivariate time series prediction and image classification examples.

show abstract

“…To this end, various architectures have been proposed with multiple reservoirs, additional projections, autoencoders, plasticity mechanisms, etc. [5,6,12,14,29,32].…”

Section: Architecturementioning

confidence: 99%

“…a time series). The MLE, denoted λ max , of a Mod-DeepESN instance can be computed for a given set of input sequences by (12)…”

Section: Lyapunov Exponentmentioning

confidence: 99%

Analysis of Wide and Deep Echo State Networks for Multiscale Spatiotemporal Time Series Forecasting

Carmichael

Syed

Kudithipudi

2019

Proceedings of the 7th Annual Neuro-Inspired Computational Elements Workshop

View full text Add to dashboard Cite

Echo state networks are computationally lightweight reservoir models inspired by the random projections observed in cortical circuitry. As interest in reservoir computing has grown, networks have become deeper and more intricate. While these networks are increasingly applied to nontrivial forecasting tasks, there is a need for comprehensive performance analysis of deep reservoirs. In this work, we study the influence of partitioning neurons given a budget and the effect of parallel reservoir pathways across different datasets exhibiting multi-scale and nonlinear dynamics.

show abstract

Deep reservoir computing: A critical experimental analysis

Cited by 373 publications

References 11 publications

Reservoir Topology in Deep Echo State Networks

Reservoir Topology in Deep Echo State Networks

The expressivity and training of deep neural networks: Toward the edge of chaos?

Analysis of Wide and Deep Echo State Networks for Multiscale Spatiotemporal Time Series Forecasting

Contact Info

Product

Resources

About