Learning in two-layered networks with correlated examples

Abstract

On-line learning in layered perceptrons is often hampered by plateaus in the time dependence of the performance. Studies on backpropagation in networks with a small number of input units have revealed that correlations between subsequently presented patterns shorten the length of such plateaus. We s h o w h o w to extend the statistical mechanics framework to quantitatively check the e ect of correlations on learning in networks with a large number of input units. The surprisingly compact description we obtain makes it possible to derive properties of on-learning with correlations directly from studies on on-line learning without correlations.

On “Natural” Learning and Pruning in Multilayered Perceptrons

“…Learning the soft-committee machine appeared to be relatively easy for all procedures, and the di erences between them are much less striking. As argued in Heskes & Coolen 1997 , the plateaus for learning the soft-committee machine and related problems studied in the statistical mechanics literature are an order of magnitude easier to tackle than the plateaus that occur when learning the tent map or the XOR function. In the former case, the plateau corresponds to a saddle point of the error: the gradient on the plateau is negligible, but the Hessian has a nite negative eigenvalue that drives the escape.…”

Quantitative Analysis of Ultrasonic B-Mode Images

“…The features were calculated in an image of 50x50 pixels and 32 gray levels, whereby the successive pixels were correlated only in the axial direction according to where r\ is a uniformly-distributed gray level between 1 and 32 and c is the correlation coefficient. 12 The correlation coefficient was systematically varied between 0 and 1 in 1,000 steps. No averaging was done; thus, in all, 1,000 images were analyzed.…”