On “Natural” Learning and Pruning in Multilayered Perceptrons

Abstract

Several studies have s h o wn that natural gradient descent for on-line learning is much more e cient than standard gradient descent. In this paper, we d e r i v e natural gradients in a slightly di erent manner and discuss implications for batch-mode learning and pruning, linking them to existing algorithms such as Levenberg-Marquardt optimization and optimal brain surgeon.The Fisher matrix plays an important role in all these algorithms. The second half of the paper discusses a layered approximation of the Fisher matrix speci c to multilayered perceptrons. Using this approximation, rather than the exact Fisher matrix, we arrive a t m uch faster natural" learning algorithms and more robust pruning procedures.

Implementing Online Natural Gradient Learning: Problems and Solutions

“…Its logarithm is (4) Hence, maximization of the conditional log-likelihood is equivalent to the minimization of the square error (5) The natural gradient learning can be described by the following updating rule: (6) where is the learning rate (7) is the inverse of the so-called Riemannian metric tensor (Fisher information matrix) of the space . It is defined as (8) where denotes expectation with respect to the density (2).…”

“…Inserting (5), the quantity can be written as (9) where denotes transposition of a vector or matrix. The problem with the natural gradient learning (6) is that the input-output density is generally unknown. Moreover you need to compute the inverse of this matrix.…”

“…Further in [1] Amari give an adaptive method of realizing natural gradient learning. Heskes [6] provided an interesting explanation of natural gradient based on an idea of [7]. The goals of this paper are summarized as follows.…”

A gradient-based algorithm competitive with variational Bayesian EM for mixture of Gaussians

“…For applications to learning MLP networks, it was proposed [5] that the dependencies between weights in different layers would be ignored, making the approximate matrix G block diagonal and hence easier to invert, since only the separate blocks need to be inverted.…”

Intrinsic plasticity via natural gradient descent with application to drift compensation

“…The concept of natural gradient has further been extended to more general classes of multidimensional regression and classification problems in [15]. An alternative derivation of the natural gradient is given in [16], together with the natural equivalent of batch learning, linked to LevenbergMarquardt optimization. Recently, the special case of learning for non-linear discriminant networks was improved by use of natural gradient in [17].…”