Pruning and regularization in reservoir computing

Dutoit, Xavier; Schrauwen, Benjamin; Campenhout, J. Van; Stroobandt, Dirk; Brussel, Hendrik Van; Nuttin, Marnix

doi:10.1016/j.neucom.2008.12.020

Cited by 97 publications

(50 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…intrinsic plasticity (Schrauwen et al, 2008b;Steil, 2007), refined training algorithms (Jaeger & Hass, 2004), leaky-integrator reservoir units (Jaeger et al, 2007a), support vector machine (Schmidhuber et al, 2007), filter neurons with delay&sum readout (Holzmann & Hauser, 2009), pruning connections within the reservoir (Dutoit et al, 2009) etc. There have also been attempts to impose specialized interconnection topolo-gies on the reservoir, e.g.…”

Section: Introductionmentioning

confidence: 99%

Simple Deterministically Constructed Cycle Reservoirs with Regular Jumps

Rodan

Tiňo

2012

Neural Computation

View full text Add to dashboard Cite

A new class of state-space models, reservoir models, with a fixed state transition structure (the "reservoir") and an adaptable readout from the state space has recently emerged as a way for time series processing/modelling. Echo State Network (ESN) is one of the simplest, yet powerful, reservoir models. ESN models are generally constructed in a randomized manner. In our previous study (Rodan & Tino, 2011) we showed that a very simple, cyclic, deterministically generated reservoirs can yield performance competitive with standard ESN. In this contribution we extend our previous study in three aspects: 1) We introduce a novel simple deterministic reservoir model, Cycle Reservoir with Jumps (CRJ), with highly constrained weight values, that has superior performance to standard ESN on a variety of temporal tasks of different origin and characteristics. 2) We elaborate on the possible link between reservoir characterizations, such as eigenvalue distribution of the reservoir matrix or pseudo-Lyapunov exponent of the input-driven reservoir dynamics, and the model performance. It has been suggested that a uniform coverage of the unit disk by such eigenvalues can lead to superior model performances. We show that despite highly constrained eigenvalue distribution, CRJ consistently outperform ESN (that have much more uniform eigenvalue coverage of the unit disk). Also, unlike in the case of ESN, pseudo-Lyapunov exponents of the selected 'optimal' CRJ models are consistently negative. 3) We present a new framework for determining short term memory capacity of linear reservoir models to a high degree of precision. Using the framework we study the effect of shortcut connections in the CRJ reservoir topology on its memory capacity.

show abstract

Section: Introductionmentioning

confidence: 99%

Simple Deterministically Constructed Cycle Reservoirs with Regular Jumps

Rodan

Tiňo

2012

Neural Computation

View full text Add to dashboard Cite

show abstract

“…Mientras que el modelo ARIMA se obtiene utilizando la función auto. arima() implementada en R del paquete forecast de [19], la cual busca el mejor modelo ARIMA para una serie de tiempo univariada; el modelo encontrado fue ARIMA(0,1,0) (2,0,2) [12]; el resultado del pronóstico se presenta también en la tabla 1 y se destaca que todos los modelos CASCOR regularizados alcanzan un error inferior al de la red CASCOR correspondiente sin regularizar. Los resultados de la tabla 1 muestran que en los modelos con tres rezagos, el CASCOR-EP+RC-1 es el que obtiene el menor error en entrenamiento y de pronóstico a 2 años, mientras que con el CASCOR-RC-1 se obtiene el menor de pronóstico a un año; no obstante, el error de pronóstico a un año del modelo CASCOR-EP+RC-1 es tan solo 4 % mayor que el menor en modelos de 3 rezagos.…”

Section: Resultados Obtenidos Y Discusiónunclassified

“…En la literatura se han escrito varios trabajos sobre cómo controlar el sobreajuste, tanto para redes neuronales (especialmente MLP) como para otros modelos de aprendizaje de máquina; entre estos se encuentran: [7][8][9][10][11][12]. En los MLP el problema de sobreajuste es controlado, principalmente, mediante tres estrategias de regularización: descomposición de pesos (weightdecay) propuesta por [8], eliminación de pesos (weight elimination) por [13] (ambas son estrategias de reducción y son descritas en [9]), y regresión en cadena (ridge regression) propuesta por [14].…”

Section: Controlando El Sobreajuste De Una Red Cascorunclassified

Control del sobreajuste en redes neuronales tipo cascada correlación aplicado a la predicción de precios de contratos de electricidad

Villa

Velásquez

Sánchez

2015

rev.ing.univ.Medellin

View full text Add to dashboard Cite

ResumenLa predicción de precios de electricidad es considerada una tarea difí-cil debido a la cantidad y complejidad de los factores que influyen en su representación, y sus relaciones. Las redes neuronales tipo cascada correlación -CASCOR-permiten, realizar un aprendizaje constructivo, capturando mejor las características de los datos; sin embargo, presentan una alta tendencia al sobreajuste. Para el control del sobreajuste en algunos ámbitos se usan técnicas de regularización. No obstante, en la literatura no existen estudios que: i) Utilicen técnicas de regularización para el control de sobreajuste en redes CASCOR; ii) Usen redes CASCOR en la predicción de series de electricidad; iii) comparen el desempeño con redes neuronales tradicionales o modelos estadísticos. El objetivo de este artículo es modelar y predecir el comportamiento de la serie de precios de contratos de electricidad en Colombia, usando redes CASCOR y controlando el sobreajuste con técnicas de regularización.Palabras clave: pronóstico de series de tiempo, redes cascada correlación, rede neuronales, mercado de electricidad colombiano

show abstract

“…Since RR does not in general yield sparse input features, it is often combined with feature selection [6][7][8]. If there are N input features to the RR algorithm and if N s and N r represent the number of selected and removed features in a forward or backward algorithm, respectively, naive implementations have a complexity of the order O(RKN N 3 s M ) and O(RKN 3 N r M ).…”

Section: Feature Selectionmentioning

confidence: 99%

“…However, many publications show the advantages of RR [6][7][8] or even linear regression [9] with sparse input features. In [7] and [9] a fast algorithm for feature selection with small datasets was proposed.…”

Section: Introductionmentioning

confidence: 99%

Optimized Parameter Search for Large Datasets of the Regularization Parameter and Feature Selection for Ridge Regression

Buteneers

Caluwaerts

Dambre

et al. 2013

Neural Process Lett

View full text Add to dashboard Cite

In this paper we propose mathematical optimizations to select the optimal regularization parameter for ridge regression using cross-validation. The resulting algorithm is suited for large datasets and the computational cost does not depend on the size of the training set. We extend this algorithm to forward or backward feature selection in which the optimal regularization parameter is selected for each possible feature set. These feature selection algorithms yield solutions with a sparse weight matrix using a quadratic cost on the norm of the weights.A naive approach to optimizing the ridge regression parameter has a computational complexity of the order O(RKN 2 M ) with R the number of applied regularization parameters, K the number of folds in the validation set, N the number of input features and M the number of data samples in the training set. Our implementation has a computational complexity of the order O(KN 3 ). This computational cost is smaller than that of regression without regularization O(N 2 M ) for large datasets and is independent of the number of applied regularization parameters and the size of the training set. Combined with a feature selection algorithm the algorithm is of complexity O(RKN N To show the performance and reduction in computational cost, we apply this technique to train recurrent neural networks using the reservoir computing approach, windowed ridge regression, least-squares support vector machines (LSSVMs) in primal space using the fixed-size LS-SVM approximation and extreme learning machines.

show abstract

Pruning and regularization in reservoir computing

Cited by 97 publications

References 20 publications

Simple Deterministically Constructed Cycle Reservoirs with Regular Jumps

Simple Deterministically Constructed Cycle Reservoirs with Regular Jumps

Control del sobreajuste en redes neuronales tipo cascada correlación aplicado a la predicción de precios de contratos de electricidad

Optimized Parameter Search for Large Datasets of the Regularization Parameter and Feature Selection for Ridge Regression

Contact Info

Product

Resources

About