Dimension of Marginals of Kronecker Product Models

Abstract

Abstract. A Kronecker product model is an exponential family whose sufficient statistics matrix factorizes as a Kronecker product of two matrices, one assigned to a visible set of variables and the other to a hidden set of variables. We estimate the dimension of the set of visible marginal probability distributions by the maximum rank of the Jacobian in the limit of large parameters. The limit is described by the tropical morphism: a piecewise linear map with pieces corresponding to slicings of the visible matrix by the normal fan of the hidden matrix. We obtain combinatorial conditions under which the model has the expected dimension, equal to the minimum of the number of natural parameters and the dimension of the ambient probability simplex. Furthermore, we prove that the binary restricted Boltzmann machine always has the expected dimension.

Restricted Boltzmann Machines: Introduction and Review

“…RBMs have been studied intensively, with tools from optimization, algebraic geometry, combinatorics, coding theory, polyhedral geometry, and information geometry among others. Some of the advances over the past few years include results in relation to their approximation properties [77,43,58,57], dimension [17,53,55], semialgebraic description [18,68], efficiency of representation [45,54], sequential optimization [23,26], statistical complexity [10], sampling and training [64,22,23,26], information geometry [7,6,41].…”

“…The dimension of the original model is simply one less. Following (23), and as discussed in [55], the Jacobian for the denormalized RBM is equivalent to the matrix with columns…”

“…The matrices for the RBM and the tropical RBM are illustrated in Figure 6. In [55] it is shown that (24) can approximate the following matrix, equivalent to the Jacobian of a mixture of m + 1 product distributions model, arbitrarily well at generic parameters:x ⊗σ (W x +c), x ∈ X .…”

“…In [17] it was conjectured that it does. The problem remains open, even though [55] gave a proof for the RBM. The description of the tropical RBM as a superposition of ReLUs could be useful here.…”

“…[55, Corollary 26]). For any n, m ∈ N the model RBM n,m , with n visible and m hidden binary units, has dimension min{2 n − 1, (n + 1)(m + 1) − 1}.…”