Analysis of Wide and Deep Echo State Networks for Multiscale Spatiotemporal Time Series Forecasting

Carmichael, Zachariah; Syed, Humza; Kudithipudi, Dhireesha

doi:10.1145/3320288.3320303

Cited by 6 publications

(4 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Further, the reservoirs in different modules could have the cross connections, shown in Figure 3, Cross DeepESN. Based on the above connection structures, Carmichael et al [109,110] designed a DeepESN with modular architecture (Mod-DeepESN), which allowed for varying topologies of deep ESNs.…”

Section: Designsmentioning

confidence: 99%

A Review of Designs and Applications of Echo State Networks

Sun¹,

Song²,

Hong³

et al. 2020

Preprint

View full text Add to dashboard Cite

Recurrent Neural Networks (RNNs) have demonstrated their outstanding ability in sequence tasks and have achieved state-of-the-art in wide range of applications, such as industrial, medical, economic and linguistic. Echo State Network (ESN) is simple type of RNNs and has emerged in the last decade as an alternative to gradient descent training based RNNs. ESN, with a strong theoretical ground, is practical, conceptually simple, easy to implement. It avoids non-converging and computationally expensive in the gradient descent methods. Since ESN was put forward in 2002, abundant existing works have promoted the progress of ESN, and the recently introduced Deep ESN model opened the way to uniting the merits of deep learning and ESNs. Besides, the combinations of ESNs with other machine learning models have also overperformed baselines in some applications. However, the apparent simplicity of ESNs can sometimes be deceptive and successfully applying ESNs needs some experience. Thus, in this paper, we categorize the ESN-based methods to basic ESNs, DeepESNs and combinations, then analyze them from the perspective of theoretical studies, network designs and specific applications. Finally, we discuss the challenges and opportunities of ESNs by summarizing the open questions and proposing possible future works.

show abstract

Section: Designsmentioning

confidence: 99%

A Review of Designs and Applications of Echo State Networks

Sun¹,

Song²,

Hong³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…The structure of the deep ESN attempts to learn multiscale temporal interactions as we saw in the hidden units for the settling pattern prediction (Carmichael et al, 2019).…”

Section: Discussionmentioning

confidence: 99%

“…The single reservoir ESN (1.4) cannot learn instantaneous relationships well (i.e., the relationship between now and the current inputs.) However, deepening and widening the ESN improves the learning of important multiscale features, which lends to its success in forecasting problems (Ma et al, 2017;Carmichael et al, 2019;McDermott and Wikle, 2019b). The widening of the ESN provides an ensemble of feedforward reservoirs.…”

Section: Leaky Integrator Esnmentioning

confidence: 99%

Alternative learning strategies for spatio-temporal processes of complex animal behavior

Schafer¹

View full text Add to dashboard Cite

The estimation of spatio-temporal dynamics of animal behavior processes is complicated by nonlinear interactions. Alternative learning methods such as machine learning, deep learning, and reinforcement learning have proven successful for approximating nonlinear system mechanisms for prediction and classification. These alternative learning frameworks can be linked to statistical models in a hierarchical framework to improve ecological inference and prediction in the presence of uncertainty. This dissertation provides three methodological extensions of alternative learning with statistical uncertainty quantification for modeling animal behavior dynamics at different scales. First, an efficient Bayesian Markov model is developed to provide inference on white-fronted geese behavior from individual accelerometer and location data while accounting for classification uncertainty. Second, nonlinear basis function expansions produced by a spatio-temporal echo state network are used as features in a hierarchical generalized linear model for predicting spatial patterns of mallard duck settling pattern counts. Lastly, Bayesian inverse reinforcement learning is developed to estimate the behavioral state costs for collective animal groups.

show abstract

“…Going towards the same direction, the Deep ESN proposed by (GALLICCHIO;MICHELI, 2016) gives a model to split the reservoir into smaller networks forming interconnected layers, analog to the different scales of brain structure. All these works show that a better performance can be achieved using a clustered network rather than a random network (DETTORI et al, 2020;CARMICHAEL;SYED;KUDITHIPUDI, 2019). Therefore, in this work, we will explore the flexibility of the reservoir by replacing the random network for a clustered network since previous works show that an improvement in the performance can be achieved with these non-random network topologies.…”

Section: List Of Tablesmentioning

confidence: 99%

Clustered Echo State networks for signal denoising and frequency filtering

Junior¹,

de²

View full text Add to dashboard Cite

Esta dissetação tem como objetivo estudar um tipo de Rede Neural Artificial (RNA), conhecido como Reservoir Computing, mais especificamente as Echo State Networks (ESNs). ESNs são redes neurais recorrentes (RNNs), que fazem o mapeamento de entrada-saída através de projeções não-lineares de alta dimensão, chamada de reservoir. No modelo clássico da ESN, a matriz das conexões internas do reservatório é usualmente uma rede aleatória Erdös-Rényi. Estudos recentes investigaram o uso de redes com clusters dentro do reservatório de uma ESN, as Clustered ESNs (CESNs), sendo que essa nova rede do reservatório apresenta uma topologia com clusters. Ambos tipos de ESNs foram aplicadas ao problema de predição de séries temporais. Neste trabalho, são propostas uma ESN com redes Barabási-Albert em cada cluster (Barabási-Albert CESN), e uma deep ESN em que cada camada dessa rede contém uma rede com clusters (Deep CESNs). Além disso, foi proposto a aplicação de ESNs e suas extensões em dois novos problemas: o filtro de frequências e a remoção de ruídos de séries temporais. Uma comparação foi feita entre o modelo clássico da ESN e suas extensões. Experimentos númericos mostram que os modelos propostos de ESNs (Barabási-Albert CESN and Deep CESNs) superam o desempenho do modelo clássico da ESN, indicando que a organização dos reservatórios em clusters ou em camadas melhoram o desempenho da rede.

show abstract

Analysis of Wide and Deep Echo State Networks for Multiscale Spatiotemporal Time Series Forecasting

Cited by 6 publications

References 33 publications

A Review of Designs and Applications of Echo State Networks

A Review of Designs and Applications of Echo State Networks

Alternative learning strategies for spatio-temporal processes of complex animal behavior

Clustered Echo State networks for signal denoising and frequency filtering

Contact Info

Product

Resources

About