Deterministic Boltzmann Learning Performs Steepest Descent in Weight-Space

Abstract

The Boltzmann machine learning procedure has been successfully applied in deterministic networks of analog units that use a mean field approximation to efficiently simulate a truly stochastic system (Peterson and Anderson 1987). This type of “deterministic Boltzmann machine” (DBM) learns much faster than the equivalent “stochastic Boltzmann machine” (SBM), but since the learning procedure for DBM's is only based on an analogy with SBM's, there is no existing proof that it performs gradient descent in any function, and it has only been justified by simulations. By using the appropriate interpretation for the way in which a DBM represents the probability of an output vector given an input vector, it is shown that the DBM performs steepest descent in the same function as the original SBM, except at rare discontinuities. A very simple way of forcing the weights to become symmetrical is also described, and this makes the DBM more biologically plausible than back-propagation (Werbos 1974; Parker 1985; Rumelhart et al. 1986).

Contrastive Hebbian Learning in the Continuous Hopfield Model

“…This pa~ per shows that Hinton's observation that CHL depends on a pe1,'formance measure [3] can be generalized to any case of the continuous Hopfieid model. Contrary to previous approaches, the derivations do not pre--sume the existence of Boltzmann machines approximated with mean field networks.…”

“…In general this procedure works well and achieves learning speeds comparable to backpropagation. There are two phenomena though that sometimes occur [3]. Occasionally the network may settle in a different attractor than the one to which it had converged in previ~ ous trials.…”

“…As we will study later, varying the relative importance of E vs. S as the activations settle may help achieve maximally harmonious (minimum E) states. In the appendix it is shown that if the activation functions are the standard (0-1) logistic, F becomes the Helmholtz free energy function as defined in [3].…”

“…Assume that, over the working region of activation states, F has a unique minimum. 3 • Thus, since the minimum is unique F(+) ~ p(-), and if J ::= 0, then on different arguments, Rinton [3] showed that CRL minimizes the equivalent of J when the activation is logistic. 3This MSumption, which is shared with Boltzmann learning, Mean Field Learning, and the Almeida-Pineda algorithm is discussed in section 4.…”

“…This modification of the Hebbian learning, first applied by Hopfield to improve the storage capacity of discrete content addressable memories without hidden units (5], appears in the Boltzmann learning algorithm (1] and its mean field approximation [11] [3]. This pa~ per shows that Hinton's observation that CHL depends on a pe1,'formance measure [3] can be generalized to any case of the continuous Hopfieid model.…”