Using random weights to train multilayer networks of hard-limiting units

Abstract

A gradient descent algorithm suitable for training multilayer feedforward networks of processing units with hard-limiting output functions is presented. The conventional backpropagation algorithm cannot be applied in this case because the required derivatives are not available. However, if the network weights are random variables with smooth distribution functions, the probability of a hard-limiting unit taking one of its two possible values is a continuously differentiable function. In the paper, this is used to develop an algorithm similar to backpropagation, but for the hard-limiting case. It is shown that the computational framework of this algorithm is similar to standard backpropagation, but there is an additional computational expense involved in the estimation of gradients. Upper bounds on this estimation penalty are given. Two examples which indicate that, when this algorithm is used to train networks of hard-limiting units, its performance is similar to that of conventional backpropagation applied to networks of units with sigmoidal characteristics are presented.

<title>Character recognition using novel optoelectronic neural network</title>

“…Threshold activation functions can be realized with analog comparators. Very few hardlimiting function, multilayer network training algorithms have been developed [15,16,20,21,22]. A multilayer extension of the LMS, known as MR2, was chosen for ease of coding [1 6].…”

Quantum generalisation of feedforward neural networks

“…the means and standard deviations of the appropriate Gaussian distributions. 33,34 In the reversible generalisation, where each neuron is replaced by a permutation matrix, we find that the output is no longer a function of the inputs and continuous weights, but rather of the inputs and a discrete set of permutation matrices. However, in the generalisation to unitaries, for a gate with n inputs and outputs, there exist an infinite number of unitaries, in contrast with the discrete set of permutation matrices.…”

Training multilayer networks with discrete activation functions

“…Bartlett in [1] introduced another approach by defining the weights as random variables with smooth distribution functions and proposed an algorithm that uses an approach that is similar to BP to adjust the parameters of the weights' distributions. In [4], Corwin suggested to train NDAs with progressively steeper analog functions to facilitate training.…”

“…Various modifications of the gradient descent have been presented to train NDAs [1,3,4,6,14,17]. However, these methods require to a certain degree, depending on the method, that the learning task should be static.…”