Reservoir Topology in Deep Echo State Networks

Abstract

Deep Echo State Networks (DeepESNs) recently extended the applicability of Reservoir Computing (RC) methods towards the field of deep learning. In this paper we study the impact of constrained reservoir topologies in the architectural design of deep reservoirs, through numerical experiments on several RC benchmarks. The major outcome of our investigation is to show the remarkable effect, in terms of predictive performance gain, achieved by the synergy between a deep reservoir construction and a structured organization of the recurrent units in each layer. Our results also indicate that a particularly advantageous architectural setting is obtained in correspondence of DeepESNs where reservoir units are structured according to a permutation recurrent matrix.

Hyperparameter tuning in echo state networks

“…The presented results differ by multiple orders of magnitude from e.g. Gallicchio and Micheli [9], who reached NARMA10 ≈ 10 −4 ± 10 −5 , MG17 ≈ 10 −9 ± 10 −10 , and MG30 ≈ 10 −8 ± 10 −9 . The parameters in the aforementioned work were tuned manually and the sparse topology ended up as the worst of the four.…”

“…Some authors (e.g., [25]) use the same constant for all the nonzero connection weights in the ring, chain, and permutation topologies instead of generating the values from N(𝜇 𝑟𝑒𝑠 , 𝜎 2 𝑟𝑒𝑠 ) as is the case for the sparse topology (e.g., [9]). In other words, the reservoir matrix 𝑊 can be expressed as 𝜆𝑊 𝑏 , where 𝑊 𝑏 is a binary matrix and 𝜆 is the desired constant.…”

“…It is worth noting that the sparse topology has O(𝑛 2 ) parameters where 𝑛 is the number of neurons, whereas the ring, chain and permutation topologies have only O(𝑛) parameters. Analogously to other papers (e.g., [9] [31]), we compare topologies with the same number of neurons, not the same number of parameters.…”

“…Unfortunately, researchers have not yet converged to a single and easily comparable performance measure and even though there exist widely used benchmark tasks, many authors have developed their specific modification or parametrization. Unless specified otherwise, we will use the measures from Gallicchio and Micheli [9]. 3.5.1 NARMA10.…”

“…The hyperparameters of each reservoir topology are optimized so that the instantiated network maximizes its performance on one of the benchmark tasks. Similarly to [9], the experiment uses ESNs with 500 reservoir neurons, regardless of the topology. The reservoir weights are generated from normal distribution N(𝜇 𝑟𝑒𝑠 , 𝜎 2 𝑟𝑒𝑠 ), feedback weights from uniform distribution 𝑈 (−𝜔 𝑓 𝑏 , 𝜔 𝑓 𝑏 ), and input weights from 𝑈 (−𝜔 𝑖𝑛 , 𝜔 𝑖𝑛 ).…”