Replica Symmetry Breaking and the Kuhn-Tucker Cavity Method in Simple and Multilayer Perceptrons

Abstract

Within a Kuhn-Tucker cavity method introduced in a former paper, we study optimal stability learning for situations, where in the replica formalism the replica symmetry may be broken, namely (i) the case of a simple perceptron above the critical loading, and (ii) the case of two-layer AND-perceptrons, if one learns with maximal stability. We find that the deviation of our cavity solution from the replica symmetric one in these cases is a clear indication of the necessity of replica symmetry breaking. In any case the cavity solution tends to underestimate the storage capabilities of the networks.

Storage capacity of a constructive learning algorithm

“…It is however worth to examine it. The corresponding value of θ as a function of ε and κ follows form (18), and the relation between α, θ, κ and c from (17). We find:…”

“…In order to circumvent the RS approximation, we determine the training error E t (α, ε) using the Kuhn-Tucker (KT) cavity method proposed by Gerl and Krey [18], that we generalize here to the case of a perceptron with a threshold learning a training set with a biased probability of targets given by (5). Contrary to the RS solution, this cavity method has been shown to overestimate the training error [18]. Consequently, the results allow us to deduce an upper bound for the number of perceptrons needed by the tilinglike procedure to converge.…”

“…As we expect that the gap vanishes for α → +∞, we look for solutions of the extremum equations with c → 0 and |θ| → +∞ (notice that θ is negative for ε < 1/2) with the product a = θ √ 2c finite. Introducing these assumptions into (18), we determine ε as a function of a:…”

“…These are deduced in section IV for the Gardner cost function with vanishing and finite margin, within the replica-symmetry (RS) approximation. As this approximation is known to provide only a lower bound to the perceptron's actual training error [16,17], we also determined an upper bound through a generalization of the Kuhn-Tucker (KT) cavity method proposed by Gerl and Krey [18]. The general expression for the number of hidden perceptrons generated by the TLA in the limit α → +∞ is deduced in section V. Our main result is that the number of hidden units needed by the TLA to converge grows proportionally to α/(ln α) ν in the large α limit, where ν = 1 in the RS approximation and ν = 0 within the KT cavity method, provided that the hidden perceptrons learn through the minimization of their training errors.…”

Storage capacity of the Tilinglike Learning Algorithm

“…It gives the lowest bound to E t . In the limit of large training set size α, we obtain: In order to obtain a lower bound to α arch c (k) we used the Kuhn-Tucker cavity method [14,11,12], which gives an upper bound to E t . As a result of both calculations, we can bound α alg c (k, GD):…”