Concentric ESN: Assessing the Effect of Modularity in Cycle Reservoirs

Bacciu, Davide; Bongiorno, Andrea

doi:10.1109/ijcnn.2018.8489462

Cited by 3 publications

(6 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the methodological side, a natural extension of the work in this paper is to analyze the effect of a broader pool of reservoir architectural variants, including e.g. small-world [19], cycles with regular jumps [24] and concentric [1] reservoirs. Moreover, future research could pursue even further the simplification of architectural construction in deep RC models, reducing the impact of randomness in the network initialization in the same vein as the works on minimum complexity ESNs [23,24].…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Reservoir Topology in Deep Echo State Networks

Gallicchio

Micheli

2019

Artificial Neural Networks and Machine Learning – ICANN 2019: Workshop and Special Sessions

View full text Add to dashboard Cite

Deep Echo State Networks (DeepESNs) recently extended the applicability of Reservoir Computing (RC) methods towards the field of deep learning. In this paper we study the impact of constrained reservoir topologies in the architectural design of deep reservoirs, through numerical experiments on several RC benchmarks. The major outcome of our investigation is to show the remarkable effect, in terms of predictive performance gain, achieved by the synergy between a deep reservoir construction and a structured organization of the recurrent units in each layer. Our results also indicate that a particularly advantageous architectural setting is obtained in correspondence of DeepESNs where reservoir units are structured according to a permutation recurrent matrix.

show abstract

Section: Discussionmentioning

confidence: 99%

“…In particular, at time-step t, the state of the first layer, i.e. x (1) (t) ∈ R NR , is computed as follows:…”

Section: Deep Echo State Networkmentioning

confidence: 99%

Reservoir Topology in Deep Echo State Networks

Gallicchio

Micheli

2019

Artificial Neural Networks and Machine Learning – ICANN 2019: Workshop and Special Sessions

View full text Add to dashboard Cite

show abstract

“…Different studies have proposed alternatives to the random structure of the connectivity matrix of ESNs, formulating models of reservoirs with regular graph structures. Examples include a delay line [17], where each node receives and provides information only from the previous node and the following one respectively, and the concentric reservoir proposed in [18], where multiple delay lines are connected to form a concentric structure. Furthermore, the idea of a hierarchical architecture of ESNs, where each ESN is connected to the preceding and following one, has attracted the reservoir computing community for its capability of discovering higher level features of the external signal [34].…”

Section: Hierarchical Echo-state Networkmentioning

confidence: 99%

“…Previous works have studied the dependence of the reservoir properties on the structure of the random connectivity adopted, studying the dependence of the reservoir performance on the parameters defining the random connectivity distribution, and formulating alternatives to the typical Erdos-Renyi graph structure of the network [16][17][18]. In this sense, in [17] a model with a regular graph structure has been proposed, where the nodes are connected forming a circular path with constant shortest path lengths equal to the size of the network, introducing long temporal memory capacity by construction.…”

Section: Introductionmentioning

confidence: 99%

Exploiting Multiple Timescales in Hierarchical Echo State Networks

Manneschi

Ellis

Gigante³

et al. 2021

Front. Appl. Math. Stat.

View full text Add to dashboard Cite

Echo state networks (ESNs) are a powerful form of reservoir computing that only require training of linear output weights while the internal reservoir is formed of fixed randomly connected neurons. With a correctly scaled connectivity matrix, the neurons’ activity exhibits the echo-state property and responds to the input dynamics with certain timescales. Tuning the timescales of the network can be necessary for treating certain tasks, and some environments require multiple timescales for an efficient representation. Here we explore the timescales in hierarchical ESNs, where the reservoir is partitioned into two smaller linked reservoirs with distinct properties. Over three different tasks (NARMA10, a reconstruction task in a volatile environment, and psMNIST), we show that by selecting the hyper-parameters of each partition such that they focus on different timescales, we achieve a significant performance improvement over a single ESN. Through a linear analysis, and under the assumption that the timescales of the first partition are much shorter than the second’s (typically corresponding to optimal operating conditions), we interpret the feedforward coupling of the partitions in terms of an effective representation of the input signal, provided by the first partition to the second, whereby the instantaneous input signal is expanded into a weighted combination of its time derivatives. Furthermore, we propose a data-driven approach to optimise the hyper-parameters through a gradient descent optimisation method that is an online approximation of backpropagation through time. We demonstrate the application of the online learning rule across all the tasks considered.

show abstract

“…Different from general RNNs trained with back-propagation, such as LSTM and gated recurrent units (GRUs), only the readout coefficients denoted by W out from the reservoir layer to the output layer need to be trained for a particular task for RC. The internal network parameters, namely the adjacency matrix W in from the input layer to the reservoir layer, and the connections inside the reservoir W are untrained, which are fixed and random [67] or in a regular topology [69][70][71]. In the training phase of conventional reservoir computing architectures, the reservoir state is collected at each discrete time step n following…”

Section: Optical Reservoir Computingmentioning

confidence: 99%

The challenges of modern computing and new opportunities for optics

Zhang

et al. 2021

PhotoniX

View full text Add to dashboard Cite

In recent years, the explosive development of artificial intelligence implementing by artificial neural networks (ANNs) creates inconceivable demands for computing hardware. However, conventional computing hardware based on electronic transistor and von Neumann architecture cannot satisfy such an inconceivable demand due to the unsustainability of Moore’s Law and the failure of Dennard’s scaling rules. Fortunately, analog optical computing offers an alternative way to release unprecedented computational capability to accelerate varies computing drained tasks. In this article, the challenges of the modern computing technologies and potential solutions are briefly explained in Chapter 1. In Chapter 2, the latest research progresses of analog optical computing are separated into three directions: vector/matrix manipulation, reservoir computing and photonic Ising machine. Each direction has been explicitly summarized and discussed. The last chapter explains the prospects and the new challenges of analog optical computing.

show abstract

Concentric ESN: Assessing the Effect of Modularity in Cycle Reservoirs

Cited by 3 publications

References 19 publications

Reservoir Topology in Deep Echo State Networks

Reservoir Topology in Deep Echo State Networks

Exploiting Multiple Timescales in Hierarchical Echo State Networks

The challenges of modern computing and new opportunities for optics

Contact Info

Product

Resources

About