Determination of the Edge of Criticality in Echo State Networks Through Fisher Information Maximization

Abstract

Abstract-It is a widely accepted fact that the computational capability of recurrent neural networks (RNNs) is maximized on the so-called "edge of criticality." Once the network operates in this configuration, it performs efficiently on a specific application both in terms of: 1) low prediction error and 2) high shortterm memory capacity. Since the behavior of recurrent networks is strongly influenced by the particular input signal driving the dynamics, a universal, application-independent method for determining the edge of criticality is still missing. In this paper, we aim at addressing this issue by proposing a theoretically motivated, unsupervised method based on Fisher information for determining the edge of criticality in RNNs. It is proved that Fisher information is maximized for (finite-size) systems operating in such critical regions. However, Fisher information is notoriously difficult to compute and requires the analytic form of the probability density function ruling the system behavior. This paper takes advantage of a recently developed nonparametric estimator of the Fisher information matrix and provides a method to determine the critical region of echo state networks (ESNs), a particular class of recurrent networks. The considered control parameters, which indirectly affect the ESN performance, are explored to identify those configurations lying on the edge of criticality and, as such, maximizing Fisher information and computational performance. Experimental results on benchmarks and real-world data demonstrate the effectiveness of the proposed method.Index Terms-Echo state network (ESN), edge of criticality, Fisher information, nonparametric estimation.

Optimal short-term memory before the edge of chaos in driven random recurrent networks

“…8). In previous work, it was argued that the maximal Fisher information can be used to detect the edge of chaos [30,31]. Our results suggest that such an approach is not necessarily effective for driven dynamical systems.…”

Echo State Networks with Self-Normalizing Activations on the Hyper-Sphere

“…When tuning the hyper-parameters of ESNs, one usually tries to bring the system close to the EoC, since it is in that region that their performance is optimal 14 . This can be explained by the fact that, when operating in that regime, the system introduces rich dynamics without denoting chaotic behavior.…”

A Characterization of the Edge of Criticality in Binary Echo State Networks

“…Various empirical results suggest that ESNs achieve the highest expressive power, i.e., the ability to provide optimal performances, exactly when configured on the edge of the transition between a ordered and chaotic regime (e.g., see [6,[9][10][11][12]15]). Once the network operates on the edge -or in proximity to -it achieves the highest memory capacity (storage of past information) and accuracy prediction, compatible with the network architecture.…”

“…Once the network operates on the edge -or in proximity to -it achieves the highest memory capacity (storage of past information) and accuracy prediction, compatible with the network architecture. For determining the edge of chaos, one usually resorts to computing the maximum Lyapunov exponent [5] or identify parameter configuration maximizing the Fisher information [10].…”