Homeostatic Plasticity for Single Node Delay-Coupled Reservoir Computing

Abstract

Supplementing a differential equation with delays results in an infinite-dimensional dynamical system. This property provides the basis for a reservoir computing architecture, where the recurrent neural network is replaced by a single nonlinear node, delay-coupled to itself. Instead of the spatial topology of a network, subunits in the delay-coupled reservoir are multiplexed in time along one delay span of the system. The computational power of the reservoir is contingent on this temporal multiplexing. Here, we learn optimal temporal multiplexing by means of a biologically inspired homeostatic plasticity mechanism. Plasticity acts locally and changes the distances between the subunits along the delay, depending on how responsive these subunits are to the input. After analytically deriving the learning mechanism, we illustrate its role in improving the reservoir's computational power. To this end, we investigate, first, the increase of the reservoir's memory capacity. Second, we predict a NARMA-10 time series, showing that plasticity reduces the normalized root-mean-square error by more than 20%. Third, we discuss plasticity's influence on the reservoir's input-information capacity, the coupling strength between subunits, and the distribution of the readout coefficients.

Persistent Memory in Single Node Delay-Coupled Reservoir Computing

“…This is because increasing the number of virtual nodes within constant delay τ also increases cross-correlations, since the delays between virtual nodes become shorter. The complexity of a DCR can only be controlled by understanding the tight interplay between the number of virtual nodes and their location, the total delay, the mask structure, and the nonlinearity responsible of mixing past and current inputs [8]. …”

“…The DCR provides a promising testing bed to theories of neural computations with delays. For instance, some of the authors have shown that applying homeostatic plasticity [7] directly to the delays dramatically improved the computational capabilities of the DCR [8]. …”

“…This procedure, called FORCE, requires chaotic dynamics in the spontaneous activity of the recurrent neural network. The current paper, however, only considers fixed point spontaneous DCR dynamics in concord with previous studies [5, 8, 24], and exploring a setup similar to FORCE will be carried out elsewhere. In addition, in its original form, a DCR only retains temporal dependencies on a short time scale.…”

“…This leads to virtual nodes , at which the dynamics of the single delay-coupled neuron is sampled. Despite these differences, a DCR is approximately equivalent to a recurrent neural network with constrained connectivity [5, 6, 8]. This translates to comparable performance [5], as well as similar principles of self-organization based on plasticity [28–30] that can improve the DCR in an unsupervised fashion [8].…”

“…Despite these differences, a DCR is approximately equivalent to a recurrent neural network with constrained connectivity [5, 6, 8]. This translates to comparable performance [5], as well as similar principles of self-organization based on plasticity [28–30] that can improve the DCR in an unsupervised fashion [8]. At the same time, the simple architecture of the DCR allows for a largely reduced complexity in physical implementation, which has already been demonstrated on electronic [5], optoelectronic [31, 32] (with the input either in synchrony with [31] or asynchronous to [32] the delay line), and ultrafast all-optical hardware [33].…”