Regularization by Intrinsic Plasticity and Its Synergies with Recurrence for Random Projection Methods

Abstract

Neural networks based on high-dimensional random feature generation have become popular under the notions extreme learning machine (ELM) and reservoir computing (RC). We provide an in-depth analysis of such networks with respect to feature selection, model complexity, and regularization. Starting from an ELM, we show how recurrent connections increase the effective complexity leading to reservoir networks. On the contrary, intrinsic plasticity (IP), a biologically inspired, unsupervised learning rule, acts as a task-specific feature regularizer, which tunes the effective model complexity. Combing both mechanisms in the framework of static reservoir computing, we achieve an excellent balance of feature complexity and regularization, which provides an impressive robustness to other model selection parameters like network size, initialization ranges, or the regularization parameter of the output learning. We demonstrate the advantages on several synthetic data as well as on benchmark tasks from the UCI repository providing practical insights how to use high-dimensional random networks for data processing

Binary/ternary extreme learning machines

“…In [5,7,6,24] the principle of intrinsic plasticity is transferred to ELMs and introduced as an efficient pretraining method, aimed at adapting the hidden layer weights and biases, such that the output distribution of the hidden layer is shaped like an exponential distribution. The only parameter of batch intrinsic plasticity is the mean μ exp of the target exponential distribution.…”

“…One approach for adapting the hidden layer to the context is the mechanism of batch intrinsic plasticity (BIP) [5][6][7]. The idea of BIP is that it adapts the slope and bias of the hidden layer neurons such that their outputs are approximately exponentially distributed.…”

Optimizing extreme learning machines via ridge regression and batch intrinsic plasticity

“…The regression parameter is α = 10 −3 in the following experiments. The interplay between IP and ELMs has been analyzed in rigorous detail in [6]. An highly efficient batch version of IP suited for ELMs was proposed in [5].…”

Modeling neural plasticity in echo state networks for time series prediction

“…multiplicative normalization, solves the stability problems and generates an algorithm for principal components analysis [26]. This is a computational form of an effect which is believed to happen in biological neurons.…”