Deep Boltzmann Machines: Rigorous Results at Arbitrary Depth

Abstract

A class of deep Boltzmann machines is considered in the simplified framework of a quenched system with Gaussian noise and independent entries. The quenched pressure of a K-layers spin glass model is studied allowing interactions only among consecutive layers. A lower bound for the pressure is found in terms of a convex combination of K Sherrington–Kirkpatrick models and used to study the annealed and replica symmetric regimes of the system. A map with a one-dimensional monomer–dimer system is identified and used to rigorously control the annealed region at arbitrary depth K with the methods introduced by Heilmann and Lieb. The compression of this high-noise region displays a remarkable phenomenon of localisation of the processing layers. Furthermore, a replica symmetric lower bound for the limiting quenched pressure of the model is obtained in a suitable region of the parameters and the replica symmetric pressure is proved to have a unique stationary point.

The Solution of the Deep Boltzmann Machine on the Nishimori Line

“…The random term in (8) corresponds to the Hamiltonian studied in [3], but the addition of a deterministic part changes the properties of the model. We denote the random pressure per particle by…”

“…Proposition 1 relies on an algebraic lemma, which we write here for convenience. Its proof can be found in [3] (see Lemma 1 therein). oo) .…”

“…Uniqueness of the solution of the consistency equation for positive external fields can be proven adapting the strategy used in [3], where the replica symmetric equation of a deep Boltzmann machines was proved to admit a unique solution when the couplings and the external fields are centred Gaussian random variables. In particular the layers structure permits to "decouple" the interactions as shown in the following Remark 7.…”

“…In fact the positivity property, encoded in an elliptic condition, requires dominant intra-group interaction with respect to inter-group ones. While the general deep (restricted) Boltzmann machine is still an unsolved problem (see nevertheless [1,[3][4][5]12,18,23,24] for centered Gaussian interactions), we present here its exact and rigorous solution in a subregion of the phase space known as Nishimori line. In a previous paper [2] we have fully solved the elliptic multi-species model on the Nishimori line, where the property of replica symmetry, i.e.…”

Learning and Retrieval Operational Modes for Three-Layer Restricted Boltzmann Machines

“…In this paper we focus on multilayer structures with only one hidden layer (namely, the networks are shallow), where connections between neurons are symmetric (namely, the networks are recurrent), and where connections only occur between neurons belonging to different layers (namely, the networks are restricted). For this kind of neural networks, also referred to as restricted Boltzmann machines (RBMs), a few rigorous results are available (see e.g., [3][4][5][6][7]). Most of these results were obtained by leveraging a formal equivalence between two-layer RBMs and Hopfield networks (HN) [8].…”