Fluctuations in the random-link matching problem

Malatesta, Enrico M.; Parisi, Giorgio; Sicuro, Gabriele

doi:10.1103/physreve.100.032102

Cited by 6 publications

(3 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Statistical mechanics predicts the properties of a macroscopic system from the laws of its microscopic dynamics [2][3][4]. In this area, a major role is played by phase transitions that regulate what is achievable in principle (information theoretical thresholds) and what is achievable in practice (algorithmic thresholds) [5][6][7][8][9][10][11][12][13]. The graphical representation (the phase diagram) of the various phases delimited by phase transitions allows researchers to predict the response of the system as a function of its own tunable parameters.…”

Section: Introductionmentioning

confidence: 99%

Phase transitions in the mini-batch size for sparse and dense two-layer neural networks

Marino,

Ricci-Tersenghi

2024

Mach. Learn.: Sci. Technol.

View full text Add to dashboard Cite

The use of mini-batches of data in training artificial neural networks is nowadays very common. Despite its broad usage, theories explaining quantitatively how large or small the optimal mini-batch size should be are missing. This work presents a systematic attempt at understanding the role of the mini-batch size in training two-layer neural networks.Working in the teacher-student scenario, with a sparse teacher, and focusing on tasks of different complexity, we quantify the effects of changing the mini-batch size $m$.We find that often the generalization performances of the student strongly depend on $m$ and may undergo sharp phase transitions at a critical value $m_c$, such that for $m<m_c$ the training process fails, while for $m>m_c$ the student learns perfectly or generalizes very well the teacher. Phase transitions are induced by collective phenomena firstly discovered in statistical mechanics and later observed in many fields of science. Observing a phase transition by varying the mini-batch size across different architectures raises several questions about the role of this hyperparameter in the neural network learning process.

show abstract

Section: Introductionmentioning

confidence: 99%

Phase transitions in the mini-batch size for sparse and dense two-layer neural networks

Marino,

Ricci-Tersenghi

2024

Mach. Learn.: Sci. Technol.

View full text Add to dashboard Cite

show abstract

“…They proved that the random-link matching problem can be fully investigated using the replica method and the cavity method in the limit of large number of vertices [14]. Exact results have been obtained about the leading order cost and the finite-size corrections [15][16][17], the fluctuation of the average optimal cost [18], and its embedding in the Euclidean space [19][20][21].…”

Section: Introductionmentioning

confidence: 99%

Random-link matching problems on random regular graphs

Parisi¹,

Perrupato²,

Sicuro³

2020

J. Stat. Mech.

Self Cite

View full text Add to dashboard Cite

We study the random-link matching problem on random regular graphs, alongside with two relaxed versions of the problem, namely the fractional matching and the so-called "loopy" fractional matching. We estimated the asymptotic average optimal cost using the cavity method. Moreover, we also study the finite-size corrections due to rare topological structures appearing in the graph at large sizes. We estimate these contributions using the cavity approach, and we compare our results with the output of numerical simulations. The analysis also clarifies the meaning of the finite-size contributions appearing in the fullyconnected version of the problem, that has been already analyzed in the literature.Random-link matching problems on random regular graphs 2 relevance. In particular, the theory of disordered systems seems to be especially suitable for the study of the typical properties of optimization problems in presence of randomness [8][9][10][11]. The exploration of this topic, that also inspired powerful numerical techniques for the algorithmic solution of these problems, started in the eighties, with the seminal works of Orland [12] and Mézard and Parisi [13], that tackled, as prototype problem, the random-link matching problem on the complete graph. They proved that the random-link matching problem can be fully investigated using the replica method and the cavity method in the limit of large number of vertices [14]. Exact results have been obtained about the leading order cost and the finite-size corrections [15][16][17], the fluctuation of the average optimal cost [18], and its embedding in the Euclidean space [19][20][21].Despite its popularity in the fully-connected version, very few results are available on the random-link matching problem on sparse topologies. Zhou and Ou-Yang [22] and subsequently Zdeborová and Mézard [23] studied, using the cavity method, the number of maximum and perfect (unweighted) matchings on sparse graphs. As far as we know, a study of the random weighted matching problem on the Bethe lattice is, instead, missing in the literature.In this paper, we investigate exactly this formulation of the matching problem using the cavity method, both at the leading order and at the level of finite-size corrections. This analysis will be, first of all, of methodological interest: indeed, we will check, for the first time to our knowledge, the effectiveness of the cavity method in the evaluation of the contribution to finitesize corrections of rare topological structures in a combinatorial optimization problem defined on a sparse graph. Such an approach, introduced in the study of spin glasses in Refs. [24][25][26], allows us to go beyond the leading order using results (like the cavity fields distribution) obtained at the leading level. Moreover, these topological contributions seem to be related to the finitesize corrections appearing in the fully-connected case, a fact suggested in Ref.[17] that will be clarified in the present paper, where this correspondence will be made explicit.The paper is ...

show abstract

“…However, as it has been argued since the early days of the subject, one can employ spin glass techniques in a more general setting, to estimate probability distributions [1] and fluctuations around the typical values [2,3] of quantities of interest. More recently, Rivoire [4], Parisi and Rizzo [5][6][7][8] and others [9][10][11] followed this line of thought, providing a bridge between spin glasses (and disordered systems more in general, as in [12]) and the theory of large deviations, that deals with rare events whose probability decays exponentially in the system size. This topic, which is the natural framework to set statistical mechanics in a mathematical perspective, has recently been the subject of a comprehensive and pedagogical review by Touchette [13], as well as of intensive efforts in non-equilibrium statistical physics [14].…”

mentioning

confidence: 99%

Large deviations of the free energy in the p -spin glass spherical model

Pastore

Gioacchino

Rotondo

2019

Phys. Rev. Research

View full text Add to dashboard Cite

We investigate the behavior of the rare fluctuations of the free energy in the p-spin spherical model, evaluating the corresponding rate function via the Gärtner-Ellis theorem. This approach requires the knowledge of the analytic continuation of the disorder-averaged replicated partition function to arbitrary real number of replicas. In zero external magnetic field, we show via a one-step replica symmetry breaking (1RSB) calculation that the rate function is infinite for fluctuations of the free energy above its typical value, corresponding to an anomalous, super-extensive suppression of rare fluctuations. We extend this calculation to non-zero magnetic field, showing that in this case this very large deviation disappears and we try to motivate this finding in light of a geometrical interpretation of the scaled cumulant generating function. I. DISORDERED SYSTEMS AND LARGE DEVIATIONSarXiv:1909.06196v3 [cond-mat.dis-nn]

show abstract

Fluctuations in the random-link matching problem

Cited by 6 publications

References 32 publications

Phase transitions in the mini-batch size for sparse and dense two-layer neural networks

Phase transitions in the mini-batch size for sparse and dense two-layer neural networks

Random-link matching problems on random regular graphs

Large deviations of the free energy in the p -spin glass spherical model

Contact Info

Product

Resources

About