Free Energy and Complexity of Spherical Bipartite Models

Auffinger, Antonio; Chen, Wei Kuo

doi:10.1007/s10955-014-1073-0

Cited by 48 publications

(55 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…When instead the interaction coefficients are on the hyperbolic regime the model is beyond the classical techniques available to solve it. The case K = 2 is known in the litterature as bipartite spin glass model and has been studied in [4,7].…”

Section: Introduction and Resultsmentioning

confidence: 99%

Annealing and Replica-Symmetry in Deep Boltzmann Machines

et al. 2020

View full text Add to dashboard Cite

In this paper we study the properties of the quenched pressure of a multi-layer spin-glass model (a deep Boltzmann Machine in artificial intelligence jargon) whose pairwise interactions are allowed between spins lying in adjacent layers and not inside the same layer nor among layers at distance larger than one. We prove a theorem that bounds the quenched pressure of such a K-layer machine in terms of K Sherrington-Kirkpatrick spin glasses and use it to investigate its annealed region. The replica-symmetric approximation of the quenched pressure is identified and its relation to the annealed one is considered. The paper also presents some observation on the model's architectural structure related to machine learning. Since escaping the annealed region is mandatory for a meaningful training, by squeezing such region we obtain thermodynamical constraints on the form factors. Remarkably, its optimal escape is achieved by requiring the last layer to scale sub-linearly in the network size.

show abstract

Section: Introduction and Resultsmentioning

confidence: 99%

Annealing and Replica-Symmetry in Deep Boltzmann Machines

et al. 2020

View full text Add to dashboard Cite

show abstract

“…The goal of this paper is to study the free energy F N 1 ,N 2 (β) = N −1 log Z N 1 ,N 2 (β) as N 1 , N 2 → ∞. For small enough β, Auffinger and Chen obtained a minimization formula for the limiting free energy in [3]. We mention that their work applies to more general mixed (p, q)-spin Hamiltonians with external fields.…”

Section: Bipartite Sskmentioning

confidence: 99%

“…Auffinger and Chen obtained the limiting free energy when β is small enough in [3] in terms of a minimization problem. Their result applies to general mixed (p, q)-spin Hamiltonians with the presence of the external field.…”

Section: Limiting Free Energymentioning

confidence: 99%

Fluctuations of the Free Energy of the Spherical Sherrington–Kirkpatrick Model

2016

View full text Add to dashboard Cite

We consider the free energy of the bipartite spherical Sherrington-Kirkpatrick model. We find the critical temperature and prove the limiting free energy for all non-critical temperature. We also show that the law of the fluctuation of the free energy converges to the Gaussian distribution when the temperature is above the critical temperature, and to the GOE Tracy-Widom distribution when the temperature is below the critical temperature. The result is universal, and the analysis is applicable to a more general setting including the case where the disorders are non-identically distributed.We use a special structure of the Hamiltonian (1.3) to study its limiting free energy and the fluctuations. Setting the matrix J = (J ij ) and considering σ and τ as column vectors, the critical points of the function f (σ, τ ) = σ T Jτ (which is a constant multiple of the Hamiltonian) subject to the constraints σ 2 = N 1 and τ 2 = N 2 satisfy the equationswhere λ 1 and λ 2 are Lagrange multipliers. These equations imply that σ is an eigenvector of the matrix JJ T , τ is an eigenvector of J T J, and λ 1 λ 2 is an eigenvalue of J T J (and also JJ T ). The matrix J is a random matrix with independent and identically distributed entries. The matrix J T J is called a (constant multiple of) sample covariance matrix (with null covariance) in statistics and also is said to belong to the Laguerre orthogonal ensemble in random matrix theory [31,22,5]. It is one of the fundamental matrices in random matrix theory. The behavior of the eigenvalues of J T J (the squared singular values of J) in the large dimension limit is well-studied.There is a more direct connection between the random matrices and the free energy. In [27,7], it was shown that the partition function of the usual SSK model can be expressed as a random single integral. In this paper, we obtain a similar result for the bipartite SSK model, but this time the random integral is a double integral; see Lemma 2.5. This random double integral involves the eigenvalues of J T J. We analyze the double integral asymptotically using the method of steepestdescent. The reason that we can apply the method of steepest-descent to the random integral is that even though the eigenvalues are random, their fluctuations about their classical locations are small. Precise estimates for the locations of the eigenvalues were obtained recently in random matrix theory. The "rigidity" estimates for the eigenvalues of J T J were proved by Pillai and Yin in 2014 [35] for the case N 1 = N 2 . The estimates for the case N 1 = N 2 follow from [2]. The rigidity estimates are also crucial in proving the universality in random matrix. They are proved for several other random matrix ensembles [21,20,13]. Our analysis is applicable to a large class of random double integrals under certain general conditions (including the rigidity condition) on a sequence of random variables. We obtain the results for the bipartite SSK model as a special case of a more general asymptotic result for random double integrals.The s...

show abstract

“…We discover phase transitions for sMBP and LQAP, which are equivalent to the discontinuities of REM and high-temperature SK (Derrida, 1981;Aizenman et al, 1987). Our results are expected (see Auffinger and Chen, 2014) to foster understanding some fundamental algorithmic complexity properties of these and other optimization problems.…”

mentioning

confidence: 56%

“…It is known that discontinuities of the free energy indicate abrupt changes in the accessibility of solutions and they are closely related to the complexity of the problems (Auffinger and Chen, 2014). We also note that such abrupt changes of macroscopic properties, also known as phase transitions, are characteristic features of various large systems (e.g.…”

mentioning

confidence: 72%

Phase Transitions in Parameter Rich Optimization Problems

Buhmann

Dumazert

Gronskiy

et al. 2017

2017 Proceedings of the Fourteenth Workshop on Analytic Algorithmics and Combinatorics (ANALCO)

View full text Add to dashboard Cite

Most real world combinatorial optimization problems are affected by noise in the input data. Search algorithms to identify "good" solutions with low costs behave like the dynamics of large disordered particle systems, e.g., random networks or spin glasses. Such solutions to noise perturbed optimization problems are characterized by Gibbs distributions when the optimization algorithm searches for typical solutions by stochastically minimizing costs. The free energy that determines the normalization of the Gibbs distribution balances cost minimization relative to entropy maximization.The problem to analytically compute the free energy of disordered systems has been known as a notoriously difficult mathematical challenge for at least half a century (Talagrand, 2003). We provide rigorous, matching upper and lower bounds on the free energy for two disordered combinatorial optimization problems of random graph instances, the sparse Minimum Bisection Problem (sMBP) and Lawler's Quadratic Assignment Problem (LQAP). These two problems exhibit phase transitions that are equivalent to the statistical behavior of Derrida's Random Energy Model (REM). Both optimization problems can be characterized as parameter rich since individual solutions depend on more parameters than the logarithm of the solution space cardinality would suggest for e.g. a coordinate representation. Noisy Combinatorial OptimizationCombinatorial optimization arises in many real world settings and these problems are often notoriously difficult to solve due to data dependent noise in the parameters. Apart from algorithmic questions -like efficient (stochastic) search for solutions with provable guarantees -more theoretical challenges, such as generalization of solutions and their typicality relate to the * jbuhmann@inf. analytical computation of various macroscopic properties (Frenk et al., 1985) like the free energy and these problems remain largely open. Especially, the free energy of the corresponding Gibbs distribution is one of those most important macroscopic parameters that often arises in the context of combinatorial optimization. For example, Vannimenus and Mézard (1984) explored the free energy properties of the traveling salesman problem. In this paper we compute the free energy for two optimization problems -sparse Minimum Bisection (sMBP) and Lawler's Quadratic Assignment (LQAP).Both, sMBP and LQAP belong to a class of optimization problems that can be formulated as follows: Let n be an integer (e.g., number of vertices in a graph, size of a matrix, number of keys in a digital tree, etc.), and S n a set of objects (e.g., a set of vertices, elements of a matrix, keys, etc). The data X denote a set of random variables that enter into the definition of an instance (e.g., weights of edges in a weighted graph). One often is interested in asymptotic behavior of the optimal values R max or R min defined aswhere C n is a set of all feasible solutions, S n (c) is a set of objects from S n belonging to the c-th feasible solution (e.g., set of edg...

show abstract

Free Energy and Complexity of Spherical Bipartite Models

Cited by 48 publications

References 25 publications

Annealing and Replica-Symmetry in Deep Boltzmann Machines

Annealing and Replica-Symmetry in Deep Boltzmann Machines

Fluctuations of the Free Energy of the Spherical Sherrington–Kirkpatrick Model

Phase Transitions in Parameter Rich Optimization Problems

Contact Info

Product

Resources

About