Intrinsic plasticity via natural gradient descent with application to drift compensation

Neumann, Klaus; Strub, Claudius; Steil, Jochen J.

doi:10.1016/j.neucom.2012.12.047

Cited by 4 publications

(8 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Adapting the conventional gradient with respect to the Riemannian metric corrects for this non-linearity, such that the distance of two parameter sets linearly transfers to the output space. This change of the gradient is termed natural gradient and leads to a substantial performance increase in the convergence rate for IP (Neumann and Steil, 2012 ; Neumann et al, 2013 ). Therefore the natural gradient is used in this paper due to these technical benefits, although the adaptation of DNFs with IP proposed in this paper in principal also works with the standard IP adaptation.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Dynamic Neural Fields with Intrinsic Plasticity

Strub

Schöner

Wörgötter

et al. 2017

Front. Comput. Neurosci.

View full text Add to dashboard Cite

Dynamic neural fields (DNFs) are dynamical systems models that approximate the activity of large, homogeneous, and recurrently connected neural networks based on a mean field approach. Within dynamic field theory, the DNFs have been used as building blocks in architectures to model sensorimotor embedding of cognitive processes. Typically, the parameters of a DNF in an architecture are manually tuned in order to achieve a specific dynamic behavior (e.g., decision making, selection, or working memory) for a given input pattern. This manual parameters search requires expert knowledge and time to find and verify a suited set of parameters. The DNF parametrization may be particular challenging if the input distribution is not known in advance, e.g., when processing sensory information. In this paper, we propose the autonomous adaptation of the DNF resting level and gain by a learning mechanism of intrinsic plasticity (IP). To enable this adaptation, an input and output measure for the DNF are introduced, together with a hyper parameter to define the desired output distribution. The online adaptation by IP gives the possibility to pre-define the DNF output statistics without knowledge of the input distribution and thus, also to compensate for changes in it. The capabilities and limitations of this approach are evaluated in a number of experiments.

show abstract

Section: Methodsmentioning

confidence: 99%

“…A natural gradient-based parameter adaptation for IP termed NIP has been derived in Neumann and Steil ( 2012 ), here only the resulting learning rules are given:…”

Section: Methodsmentioning

confidence: 99%

Dynamic Neural Fields with Intrinsic Plasticity

Strub

Schöner

Wörgötter

et al. 2017

Front. Comput. Neurosci.

View full text Add to dashboard Cite

show abstract

“…The "wicked-map problem" appears, since it is unclear whether the Euclidean distance in the (a, b)-plane is the most natural way to measure distances between the output distributions f y that are parametrized by a and b. In fact, in 2013 a different dynamics has been predicted for the same cost function, but under the assumption of the Fisher information metric 1 [22] which can be considered a more natural choice to measure distances between dis-1 See Section 4 for more details on the Fisher metric.…”

Section: Gradient Descent In Neurosciencementioning

confidence: 99%

On the choice of metric in gradient-based theories of brain function

et al. 2020

View full text Add to dashboard Cite

The idea that the brain functions so as to minimize certain costs pervades theoretical neuroscience. Since a cost function by itself does not predict how the brain finds its minima, additional assumptions about the optimization method need to be made to predict the dynamics of physiological quantities. In this context, steepest descent (also called gradient descent) is often suggested as an algorithmic principle of optimization potentially implemented by the brain. In practice, researchers often consider the vector of partial derivatives as the gradient. However, the definition of the gradient and the notion of a steepest direction depend on the choice of a metric. Since the choice of the metric involves a large number of degrees of freedom, the predictive power of models that are based on gradient descent must be called into question, unless there are strong constraints on the choice of the metric. Here we provide a didactic review of the mathematics of gradient descent, illustrate common pitfalls of using gradient descent as a principle of brain function with examples from the literature and propose ways forward to constrain the metric.

show abstract

“…In [5,7,6,24] the principle of intrinsic plasticity is transferred to ELMs and introduced as an efficient pretraining method, aimed at adapting the hidden layer weights and biases, such that the output distribution of the hidden layer is shaped like an exponential distribution. The only parameter of batch intrinsic plasticity is the mean μ exp of the target exponential distribution.…”

Section: Batch Intrinsic Plasticity (Bip) Elmmentioning

confidence: 99%

Binary/ternary extreme learning machines

Heeswijk

Miché

2015

Neurocomputing

View full text Add to dashboard Cite

a b s t r a c tIn this paper, a new hidden layer construction method for Extreme Learning Machines (ELMs) is investigated, aimed at generating a diverse set of weights. The paper proposes two new ELM variants: Binary ELM, with a weight initialization scheme based on f0; 1g-weights; and Ternary ELM, with a weight initialization scheme based on fÀ1; 0; 1g-weights. The motivation behind this approach is that these features will be from very different subspaces and therefore each neuron extracts more diverse information from the inputs than neurons with completely random features traditionally used in ELM. Therefore, ideally it should lead to better ELMs. Experiments show that indeed ELMs with ternary weights generally achieve lower test error. Furthermore, the experiments show that the Binary and Ternary ELMs are more robust to irrelevant and noisy variables and are in fact performing implicit variable selection. Finally, since only the weight generation scheme is adapted, the computational time of the ELM is unaffected, and the improved accuracy, added robustness and the implicit variable selection of Binary ELM and Ternary ELM come for free.

show abstract

Intrinsic plasticity via natural gradient descent with application to drift compensation

Cited by 4 publications

References 26 publications

Dynamic Neural Fields with Intrinsic Plasticity

Dynamic Neural Fields with Intrinsic Plasticity

On the choice of metric in gradient-based theories of brain function

Binary/ternary extreme learning machines

Contact Info

Product

Resources

About