Performance of analog neural networks subject to drifting targets and noise

Abstract

This paper analyzes the tracking behavior of analog neurons when subjected to additive noise and slowly drifting target weights. The performance of single analog neurons and multilayer analog neural networks are studied. We consider a system identification model with a target and tracking neural network. The target weights are described by a stochastic difference equation with weights changing slowly with time. The tracker weights follow the Least Mean Square (LMS) gradient descent algorithm and at each update are given a noise corrupted value of the output of the target network. When inputs are Gaussian and the activation used is the Gaussian error function, g(z) = l/&JTz e-lai2dt. (closely approximates the standard sigmoidal activation function) the analysis is tractable. The dynamies of target and tracking networks are described by a set of stochastic difference equations. We obtain an approximation of the mean squared generalization error by linearizing the nonlinear difference equations and using simple probabilistic arguments. Analytical results are then confirmed with simulation results.

Just-in-Time Adaptive Classifiers—Part II: Designing the Classifier

“…A time adaptive self-organizing map (SOM), automatically adjusting the network parameters to work both in stationary and nonstationary environments, is presented in [11]; Carpenter et al [12] propose a fuzzy ARTMAP as a nonparametric probability estimator for nonstationary pattern recognition problems and [13] a probabilistic neural network for classifying patterns characterized by time-varying distributions. In [14], an online tracking of analog neurons subject to slowly drifting weights is proposed, while Kuh [15] provides a comparison of tracking algorithms for single-layer threshold networks in presence of random drifts.…”