Evaluating the RBM without integration using PDF projection

Abstract

Abstract-In this paper, we apply probability density function (PDF) projection to arrive at an exact closed-form expression for the marginal distribution of the visible data of a restricted Boltzmann machine (RBM) without requiring integrating over the distribution of the hidden variables or needing to know the partition function. We express the visible data marginal as a projected PDF based on a set of sufficient statistics. When a Gaussian mixture model (GMM) is used to estimate the PDF of the sufficient statistics, then we arrive at a combined RBM/GMM model that serves as a general-purpose PDF estimator and Bayesian classifier. The approach extends recusively to compute the input distribution of a multi-layer network. We demonstrate the method using a reduced subset of the MNIST handwritten character data set.

A Neural Network Based on First Principles

“…In other words, if z applied to γ −1 is derived from the second layer and not from the forward path of the first layer, then h = γ −1 (z) is not guaranteed to exist. This is the sampling efficiency issue on projected belief networks [21]. It has been experimentally shown that as a PBN is trained, the sampling efficiency approaches 1.0 [21].…”

“…This is the sampling efficiency issue on projected belief networks [21]. It has been experimentally shown that as a PBN is trained, the sampling efficiency approaches 1.0 [21].…”

Beyond Moments: Extending the Maximum Entropy Principle to Feature Distribution Constraints

“…In this case, is no longer Irwin-Hall and in fact is not available in closed-form. However, the moment-generating function is available in closed-form so the saddle point approximation may be used (see Appendix in [ 13 ]). Samples of are drawn by drawing a sample from and then sampling uniformly in the set .…”

“…Note that the term “approximation” is misleading because the SPA approximates the shape of the MGF on a contour, not the absolute value, so the SPA expression for remains very accurate, in the far tails, even when itself cannot be evaluated in machine precision. Examples of this include general linear transformations of exponential and chi-squared random variables (see Section III.C and Section IV in [ 11 ]), general linear transformations of uniform random variables (Appendix in [ 13 ]), a set of linear-quadratic forms [ 14 ], and order statistics [ 15 ].…”

“…The primary numerical difficulty in implementing ( 8 ) is the calculation of . Solutions for many of the most useful feature transformations are available [ 9 , 11 , 12 , 13 ]. However, in many real-world applications, such as neural networks, the feature transformations cannot be easily written in a compact form .…”