Orthogonality in Additive Echo State Networks

Abstract

Reservoir computing (RC) is a state-of-the-art approach for efficient training in temporal domains. In this paper, we explore new RC architectures that generalise the popular leaky echo state network model (leaky-ESN) introducing an additive orthogonal term outside the nonlinear part of the ESN equation. We investigate the benefits of employing orthogonal matrices in ESNs both inside the nonlinearity and outside of it. We show empirically how to boost the memory capacity towards the theoretical maximum value while still preserving the power of nonlinear computations. Ergo, we optimise the compromise between computing with memory and computing with nonlinearity. The proposed model demonstrates to outperform both leaky-ESN and orthogonal reservoir ESN models on tasks requiring nonlinear computations with memory.

Random orthogonal additive filters: a solution to the vanishing/exploding gradient of deep neural networks

“…This might negatively affect the expressive power of the model. Further comparative experiments of identity vs random orthogonal can be found in these preprints [54], [55], from the point of view of reservoir computing.…”