Training Recurrent Networks by Evolino

Schmidhuber, Jürgen; Wierstra, Daan; Gagliolo, Matteo; Gomez, Faustino

doi:10.1162/neco.2007.19.3.757

Cited by 201 publications

(115 citation statements)

References 28 publications

Supporting

Mentioning

107

Contrasting

Unclassified

Order By: Relevance

“…A link has often been made between Reservoir Computing and kernel machines (Schmidhuber et al, 2007;Shi and Han, 2007), since both techniques essentially map the input data to a high-dimensional space, called feature-space, in which classification or regression is then performed linearly. In the case of reservoirs, this mapping is performed explicitly, as the hidden state of the reservoir is mapped directly onto the output.…”

Section: Introductionmentioning

confidence: 99%

Recurrent Kernel Machines: Computing with Infinite Echo State Networks

Hermans

Schrauwen

2012

Neural Computation

View full text Add to dashboard Cite

Echo State Networks are large, random recurrent neural networks with a single trained linear readout layer. Despite the untrained nature of the recurrent weights, they are capable of performing universal computations on temporal input data, which makes them interesting for both theoretical research and practical applications. The key to their success lies in the fact that the network computes a broad set of nonlinear, spatiotemporal mappings of the input data, on which linear regression or classification can easily be performed. One could consider the reservoir as a spatiotemporal kernel, in which the mapping to a high-dimensional space is computed explicitly. In this paper, we build on this idea and extend the concept of ESNs to infinite sized recurrent neural networks, which can be considered as recursive kernels that subsequently can be used to create recursive SVMs. We present the theoretical framework, provide several practical examples of recursive kernels, and apply them to typical temporal tasks.

show abstract

Section: Introductionmentioning

confidence: 99%

Recurrent Kernel Machines: Computing with Infinite Echo State Networks

Hermans

Schrauwen

2012

Neural Computation

View full text Add to dashboard Cite

show abstract

“…• y 2 : MSE=3.57E-8, NRMSE=1.88E-4, using 13 SV; • y 3 : MSE=2.33E-6, NRMSE=1.23E-3, using 15 SV; • y 4 : MSE=1.75E-5, NRMSE=2.86E-3, using 15 SV; • y 5 : MSE=9.16E-5, NRMSE=5.94E-3, using 15 SV, which are considerably better than the errors reported in (Xue, Yang & Haykin, 2007) and (Schmidhuber, Wierstra, Gagliolo & Gomez, 2007) but still the orders of magnitude worse than the errors obtained by the corresponding optimized SOPNN-LM model shown in Table 3. Also the computational complexity (Maric & Ivek, 2011) of the SVM model with 13 or 15 support vectors is much higher than the complexity of the corresponding SOPNN-LM model with 3, 8, 13 or 16 2-dimensional, 2 nd order node-polynomials (1).…”

Section: Tablementioning

confidence: 86%

“…We analyze the performances of SOPNN on multiple superimposed oscillations (MSO) task that has already been attempted with varying degrees of success by using ANN and SVM (Xue, Yang & Haykin, 2007;Schmidhuber, Wierstra, Gagliolo & Gomez, 2007;Holzmann & Hauser, 2010;Ceperic, Gielen & Baric, 2012). The following multiple sinusoids are modeled: where n=1,…,700.…”

Section: Multiple Superimposed Oscillations Modelingmentioning

confidence: 99%

Optimization of self-organizing polynomial neural networks

Marić

2013

Expert Systems with Applications

View full text Add to dashboard Cite

The main disadvantage of self-organizing polynomial neural networks (SOPNN) automatically structured and trained by the group method of data handling (GMDH) algorithm is a partial optimization of model weights as the GMDH algorithm optimizes only the weights of the topmost (output) node. In order to estimate to what extent the approximation accuracy of the obtained model can be improved the particle swarm optimization (PSO) has been used for the optimization of weights of all node-polynomials. Since the PSO is generally computationally expensive and time consuming a more efficient Levenberg-Marquardt (LM) algorithm is adapted for the optimization of the SOPNN. After it has been optimized by the LM algorithm the SOPNN outperformed the corresponding models based on artificial neural networks (ANN) and support vector method (SVM). The research is based on the meta-modeling of the thermodynamic effects in fluid flow measurements with time-constraints. The outstanding characteristics of the optimized SOPNN models are also demonstrated in learning the recurrence relations of multiple superimposed oscillations (MSO). KeywordsPolynomial neural networks GMDH Levenberg-Marquardt algorithm Particle swarm optimization Time series modeling 3

show abstract

“…There exist reservoir computing networks that make use of evolutionary computation for training the weights of the reservoir, such as Evolino [144], and several other models are currently proposed, with or without spiking neurons [40,76,75]. Although the research area is in rapid expansion, several papers [175,151,94] propose valuable surveys.…”

Section: Related Reservoir Computing Workmentioning

confidence: 99%

Computing with Spiking Neuron Networks

Paugam-Moisy

Bohté

2012

Handbook of Natural Computing

202

114

View full text Add to dashboard Cite

Spiking Neuron Networks (SNNs) are often referred to as the 3 rd generation of neural networks. Highly inspired from natural computing in the brain and recent advances in neurosciences, they derive their strength and interest from an accurate modeling of synaptic interactions between neurons, taking into account the time of spike firing. SNNs overcome the computational power of neural networks made of threshold or sigmoidal units. Based on dynamic event-driven processing, they open up new horizons for developing models with an exponential capacity of memorizing and a strong ability to fast adaptation. Today, the main challenge is to discover efficient learning rules that might take advantage of the specific features of SNNs while keeping the nice properties (general-purpose, easy-to-use, available simulators, etc.) of traditional connectionist models. This chapter relates the history of the "spiking neuron" in Section 1 and summarizes the most currently-in-use models of neurons and synaptic plasticity in Section 2. The computational power of SNNs is addressed in Section 3 and the problem of learning in networks of spiking neurons is tackled in Section 4, with insights into the tracks currently explored for solving it. Finally, Section 5 discusses application domains, implementation issues and proposes several simulation frameworks.

show abstract

Training Recurrent Networks by Evolino

Cited by 201 publications

References 28 publications

Recurrent Kernel Machines: Computing with Infinite Echo State Networks

Recurrent Kernel Machines: Computing with Infinite Echo State Networks

Optimization of self-organizing polynomial neural networks

Computing with Spiking Neuron Networks

Contact Info

Product

Resources

About