Analysis and Design of Echo State Networks

Abstract

The design of echo state network (ESN) parameters relies on the selection of the maximum eigenvalue of the linearized system around zero (spectral radius). However, this procedure does not quantify in a systematic manner the performance of the ESN in terms of approximation error. This article presents a functional space approximation framework to better understand the operation of ESNs and proposes an information-theoretic metric, the average entropy of echo states, to assess the richness of the ESN dynamics. Furthermore, it provides an interpretation of the ESN dynamics rooted in system theory as families of coupled linearized systems whose poles move according to the input signal dynamics. With this interpretation, a design methodology for functional approximation is put forward where ESNs are designed with uniform pole distributions covering the frequency spectrum to abide by the richness metric, irrespective of the spectral radius. A single bias parameter at the ESN input, adapted with the modeling error, configures the ESN spectral radius to the input-output joint space. Function approximation examples compare the proposed design methodology versus the conventional design.

Investigating Echo-State Networks Dynamics by Means of Recurrence Analysis

“…It is well-known that RNNs (and then also ESNs) are effective if their internal dynamical patterns are sufficiently rich. In the literature, this is usually expressed in terms of entropy of neuron activations [22], explaining the relation between SWRP and γ .…”

“…In the former case, the output of the system is considered to design the reservoir and tune the hyperparameters of the network. In unsupervised approaches, instead, the reservoir is controlled by considering algebraic/topological properties of the weight matrix [19]- [21] or criteria based on statistics of the neuron activations [22], [23]. Recently [24], [25], it was shown that topologies designed by the following deterministic criteria produced the state-of-the-art results in all major benchmarks, while having a more contractive dynamics with respect to randomly generated reservoirs.…”

Memory in reservoirs for high dimensional input

“…Increasing the input scaling factor aggravates the memory deterioration described above. This is due to the fact that the reservoir states are pushed into the saturating parts of their activation function, which decreases the 'effective' spectral radius (the spectral radius of the Jacobian [13], [14]). …”

Computing with transiently stable states

“…In practice, W is chosen to be a random sparse matrix, to avoid too much dependency (correlation) between the states. Designing the W matrix using an information theoretic criterion on the states has been proposed in [5], yet this approach also obeys the echo state condition. A simple demonstration of the typical response of an ESN with echo state condition is given in Fig.…”

“…According to [5], the echo states form a set of basis functions that are derived dynamically from the history of the input signal. The instantaneous readout linearly combines these basis functions in order to generate a given desired response.…”