Boltzmann Machines Reduction by High-Order Decimation

Abstract

Decimation is a common technique in statistical physics that is used in the context of Boltzmann machines (BMs) to drastically reduce the computational cost at the learning stage. Decimation allows to analytically evaluate quantities that should otherwise be statistically estimated by means of Monte Carlo (MC) simulations. However, in its original formulation, this method could only be applied to restricted topologies corresponding to sparsely connected neural networks. In this brief, we present a generalization of the decimation process and prove that it can be used on any BM, regardless of its topology and connectivity. We solve the Monk problem with this algorithm and show that it performs as well as the best classification methods currently available.

Weighted contrastive divergence

“…The Parity problem is known to be very difficult to learn with classical neural networks [26,27]. It is easy to understand that this is a very difficult problem to learn in the context of Boltzmann Machines as in a high order model it requires a single weight connecting all units simultaneously [28]. This problem was tested with 8 and 10 input variables (P08 and P10, respectively).…”