Improving reservoirs using intrinsic plasticity

Schrauwen, Benjamin; Wardermann, Marion; Verstraeten, David; Steil, Jochen J.; Stroobandt, Dirk

doi:10.1016/j.neucom.2007.12.020

Cited by 198 publications

(135 citation statements)

References 23 publications

Supporting

Mentioning

130

Contrasting

Unclassified

Order By: Relevance

“…In this section we test and compare our simple CRJ reservoir topology with standard ESN and SCR on a variety of timeseries tasks widely used in the ESN literature and covering a wide spectrum of memory structure (Schrauwen et al, 2008b;Cernansky & Tino, 2008;Jaeger, 2001Jaeger, , 2002aJaeger, , 2003Jaeger & Hass, 2004;Verstraeten et al, 2007;Steil, 2007).…”

Section: Methodsmentioning

confidence: 99%

“…intrinsic plasticity (Schrauwen et al, 2008b;Steil, 2007), refined training algorithms (Jaeger & Hass, 2004), leaky-integrator reservoir units (Jaeger et al, 2007a), support vector machine (Schmidhuber et al, 2007), filter neurons with delay&sum readout (Holzmann & Hauser, 2009), pruning connections within the reservoir (Dutoit et al, 2009) etc. There have also been attempts to impose specialized interconnection topolo-gies on the reservoir, e.g.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Simple Deterministically Constructed Cycle Reservoirs with Regular Jumps

Rodan

Tiňo

2012

Neural Computation

View full text Add to dashboard Cite

A new class of state-space models, reservoir models, with a fixed state transition structure (the "reservoir") and an adaptable readout from the state space has recently emerged as a way for time series processing/modelling. Echo State Network (ESN) is one of the simplest, yet powerful, reservoir models. ESN models are generally constructed in a randomized manner. In our previous study (Rodan & Tino, 2011) we showed that a very simple, cyclic, deterministically generated reservoirs can yield performance competitive with standard ESN. In this contribution we extend our previous study in three aspects: 1) We introduce a novel simple deterministic reservoir model, Cycle Reservoir with Jumps (CRJ), with highly constrained weight values, that has superior performance to standard ESN on a variety of temporal tasks of different origin and characteristics. 2) We elaborate on the possible link between reservoir characterizations, such as eigenvalue distribution of the reservoir matrix or pseudo-Lyapunov exponent of the input-driven reservoir dynamics, and the model performance. It has been suggested that a uniform coverage of the unit disk by such eigenvalues can lead to superior model performances. We show that despite highly constrained eigenvalue distribution, CRJ consistently outperform ESN (that have much more uniform eigenvalue coverage of the unit disk). Also, unlike in the case of ESN, pseudo-Lyapunov exponents of the selected 'optimal' CRJ models are consistently negative. 3) We present a new framework for determining short term memory capacity of linear reservoir models to a high degree of precision. Using the framework we study the effect of shortcut connections in the CRJ reservoir topology on its memory capacity.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Simple Deterministically Constructed Cycle Reservoirs with Regular Jumps

Rodan

Tiňo

2012

Neural Computation

View full text Add to dashboard Cite

show abstract

“…However, as the reservoir without output feedback has a limited short-term memory [7], the time series can not be indefinitely long. Furthermore, future work will research methods to optimize the reservoir dynamics (e.g., Intrinsic Plasticity [10]) in order to generate even better features. In the context of the real-world task of NTL detection, the next crucial step is to consider additional input variables such as user neighborhood, type of connection, etc.…”

Section: Resultsmentioning

confidence: 99%

Recurrent Dynamical Projection for Time Series-Based Fraud Detection

Antonelo

State

2017

Artificial Neural Networks and Machine Learning – ICANN 2017

View full text Add to dashboard Cite

Abstract. A Reservoir Computing approach is used in this work for generating a rich nonlinear spatial feature from the dynamical projection of a limited-size input time series. The final state of the Recurrent neural network (RNN) forms the feature subsequently used as input to a regressor or classifier (such as Random Forest or Least Squares). This proposed method is used for fraud detection in the energy distribution domain, namely, detection of non-technical loss (NTL) using a real-world dataset containing only the monthly energy consumption time series of (more than 300K) users. The heterogeneity of user profiles is dealt with a clustering approach, where the cluster id is also input to the classifier. Experimental results shows that the proposed recurrent feature generator is able to extract relevant nonlinear transformations of the raw time series without a priori knowledge and perform as good as (and sometimes better than) baseline models with handcrafted features.

show abstract

“…However, a more biologically realistic model would also contain some form of learning within the reservoir such as Spike-Timing Dependent Plasticity (STDP). The effects of intrinsic plasticity on the properties of reservoirs have been studied by Steil [5] and Schrauwen et al [6]. Triesch [7] has also studied the effects of intrinsic plasiticity and its synergies with synaptic plasticity in reservoirs.…”

Section: Introductionmentioning

confidence: 99%

Improved spike-timed mappings using a tri-phasic spike timing-dependent plasticity rule

Grüning

2012

The 2012 International Joint Conference on Neural Networks (IJCNN)

View full text Add to dashboard Cite

Abstract-Reservoir computing and the liquid state machine model have received much attention in the literature in recent years. In this paper we investigate using a reservoir composed of a network of spiking neurons, with synaptic delays, whose synapses are allowed to evolve using a tri-phasic spike timingdependent plasticity (STDP) rule. The networks are trained to produce specific spike trains in response to spatio-temporal input patterns. The results of using a tri-phasic STDP rule on the network properties are compared to those found using the more common exponential form of the rule. It is found that each rule causes the synaptic weights to evolve in significantly different fashions giving rise to different network dynamics. It is also found that the networks evolved with the tri-phasic rule are more capable of mapping input spatio-temporal patterns to the output spike trains.

show abstract

Improving reservoirs using intrinsic plasticity

Cited by 198 publications

References 23 publications

Simple Deterministically Constructed Cycle Reservoirs with Regular Jumps

Simple Deterministically Constructed Cycle Reservoirs with Regular Jumps

Recurrent Dynamical Projection for Time Series-Based Fraud Detection

Improved spike-timed mappings using a tri-phasic spike timing-dependent plasticity rule

Contact Info

Product

Resources

About