In this paper we continue our investigation of the analogical neural network, by introducing and studying its replica symmetric approximation in the absence of external fields. Bridging the neural network to a bipartite spin-glass, we introduce and apply a new interpolation scheme to its free energy, that naturally extends the interpolation via cavity fields or stochastic perturbations from the usual spin glass case to these models. While our methods allow the formulation of a fully broken replica symmetry scheme, in this paper we limit ourselves to the replica symmetric case, in order to give the basic essence of our interpolation method. The order parameters in this case are given by the assumed averages of the overlaps for the original spin variables, and for the new Gaussian variables. As a result, we obtain the free energy of the system as a sum rule, which, at least at the replica symmetric level, can be solved exactly, through a self-consistent mini-max variational principle. The so gained replica symmetric approximation turns out to be exactly correct in the ergodic region, where it coincides with the annealed expression for the free energy, and in the low density limit of stored patterns. Moreover, in the spin glass limit it gives the correct expression for the replica symmetric approximation in this case. We calculate also the entropy density in the low temperature region, where we find that it becomes negative, as expected for this kind of approximation. Interestingly, in contrast with the case where the stored patterns are digital, no phase transition is found in the low temperature limit, as a function of the density of stored patterns

Aim of this paper is to give an extensive treatment of bipartite mean field spin systems, ordered and disordered: at first, bipartite ferromagnets are investigated, achieving an explicit expression for the free energy trough a new minimax variational principle. Furthermore via the Hamilton-Jacobi technique the same free energy structure is obtained together with the existence of its thermodynamic limit and the minimax principle is connected to a standard max one. The same is investigated for bipartite spin-glasses: By the Borel-Cantelli lemma a control of the high temperature regime is obtained, while via the double stochastic stability technique we get also the explicit expression of the free energy at the replica symmetric level, uniquely defined by a minimax variational principle again. A general results that states that the free energies of these systems are convex linear combinations of their independent one party model counterparts is achieved too. For the sake of completeness we show further that at zero temperature the replica symmetric entropy becomes negative and, consequently, such a symmetry must be broken. The treatment of the fully broken replica symmetry case is deferred to a forthcoming paper. As a first step in this direction, we start deriving the linear and quadratic constraints to overlap fluctuations.

Restricted Boltzmann machines are described by the Gibbs measure of a bipartite spin glass, which in turn can be seen as a generalized Hopfield network. This equivalence allows us to characterize the state of these systems in terms of their retrieval capabilities, both at low and high load, of pure states. We study the paramagnetic-spin glass and the spin glass-retrieval phase transitions, as the pattern (i.e., weight) distribution and spin (i.e., unit) priors vary smoothly from Gaussian real variables to Boolean discrete variables. Our analysis shows that the presence of a retrieval phase is robust and not peculiar to the standard Hopfield model with Boolean patterns. The retrieval region becomes larger when the pattern entries and retrieval units get more peaked and, conversely, when the hidden units acquire a broader prior and therefore have a stronger response to high fields. Moreover, at low load retrieval always exists below some critical temperature, for every pattern distribution ranging from the Boolean to the Gaussian case.

We study Generalised Restricted Boltzmann Machines with generic priors for units and weights, interpolating between Boolean and Gaussian variables. We present a complete analysis of the replica symmetric phase diagram of these systems, which can be regarded as Generalised Hopfield models. We underline the role of the retrieval phase for both inference and learning processes and we show that retrieval is robust for a large class of weight and unit priors, beyond the standard Hopfield scenario. Furthermore we show how the paramagnetic phase boundary is directly related to the optimal size of the training set necessary for good generalisation in a teacher-student scenario of unsupervised learning.

Inspired by the bridge pioneered by Guerra among statistical mechanics on lattice and analytical mechanics on 1+1 continuous Euclidean space time, we built a self-consistent method to solve for the thermodynamics of mean field models defined on lattice, whose order parameters self-average. We show the whole procedure by analyzing in full detail the simplest test case, namely, the Curie-Weiss model. Further, we report some applications also to models whose order parameters do not self-average by using the Sherrington-Kirkpatrick spin glass as a guide