Scaling description of generalization with number of parameters in deep learning

Geiger, Mario; Jacot, Arthur Paul; Spigler, Stefano; Gabriel, Franck; Sagun, Levent; d’Ascoli, Stéphane; Biroli, Giulio; Hongler, Clément; Wyart, Matthieu

doi:10.1088/1742-5468/ab633c

Cited by 108 publications

(114 citation statements)

References 57 publications

Supporting

Mentioning

104

Contrasting

Order By: Relevance

“…Beyond the jamming point, the accuracy keeps steadily improving as the number of parameters increases [25,26,27], although it does so quite slowly. We have provided a quantitative explanation for this phenomenon in [33]. In .…”

Section: Generalization At and Beyond Jammingmentioning

confidence: 77%

“…Further theoretical studies on regression showed a precise mathematical description of the cusp behaviour in [30,31,32], albeit on models that are practically somewhat further away from modern neural networks. Finally, in [33], our subsequent work, we develop a quantitative theory for (i) the cusp which is associated to the divergence of the norm of the output function at the critical point of the phase transition and (ii) the asymptotic behaviour of the generalization error as N → ∞ which is associated with the reduced fluctuations of the output function. That work also shows that after ensemble averaging several networks, performance is optimal near the jamming the threshold, emphasizing the practical importance of this transition.…”

Section: Generalization Versus Over-fittingmentioning

confidence: 99%

See 1 more Smart Citation

A jamming transition from under- to over-parametrization affects generalization in deep learning

Spigler¹,

Geiger²,

d’Ascoli³

et al. 2019

J. Phys. A: Math. Theor.

View full text Add to dashboard Cite

In this paper we first recall the recent result that in deep networks a phase transition, analogous to the jamming transition of granular media, delimits the over-and under-parametrized regimes where fitting can or cannot be achieved. The analysis leading to this result support that for proper initialization and architectures, in the whole over-parametrized regime poor minima of the loss are not encountered during training, because the number of constraints that hinders the dynamics is insufficient to allow for the emergence of stable minima. Next, we study systematically how this transition affects generalization properties of the network (i.e. its predictive power). As we increase the number of parameters of a given model, starting from an under-parametrized network, we observe for gradient descent that the generalization error displays three phases: (i) initial decay, (ii) increase until the transition point -where it displays a cusp -and (iii) slow decay toward an asymptote as the network width diverges. However if early stopping is used, the cusp signaling the jamming transition disappears. Thereby we identify the region where the classical phenomenon of over-fitting takes place as the vicinity of the jamming transition, and the region where the model keeps improving with increasing the number of parameters, thus organizing previous empirical observations made in modern neural networks. ‡ The often used cross-entropy loss function also displays a transition where all data are well-fitted. However, in the over-parametrized regime the dynamic never stops, as the total loss vanishes only if the output and therefore the weights diverge. Imposing a time cut-off is done in practice, but it blurs the criticality near jamming, as exemplified below with the early stopping procedure.

show abstract

Section: Generalization At and Beyond Jammingmentioning

confidence: 77%

Section: Generalization Versus Over-fittingmentioning

confidence: 99%

A jamming transition from under- to over-parametrization affects generalization in deep learning

Spigler¹,

Geiger²,

d’Ascoli³

et al. 2019

J. Phys. A: Math. Theor.

View full text Add to dashboard Cite

show abstract

“…In practice, users stop learning at finite times (which is not needed for the hinge loss where the dynamics really stops in the overparametrized regime when the loss vanishes). Working at finite time however blurs true critical behavior near jamming, as discussed in [47].…”

Section: Cross-entropy Lossmentioning

confidence: 99%

“…It would be interesting to investigate whether the learning dynamics, at intermediate times where many data are not fitted yet, resembles the dynamics near threshold and displays bursts of changes in the constraints. Concerning the latter, we have studied the effect of jamming on generalization since this article was first written, as appears in [46,47]. 10.…”

Section: Role Of Depthmentioning

confidence: 99%

See 1 more Smart Citation

Jamming transition as a paradigm to understand the loss landscape of deep neural networks

et al. 2019

Self Cite

View full text Add to dashboard Cite

Deep learning has been immensely successful at a variety of tasks, ranging from classification to artificial intelligence. Learning corresponds to fitting training data, which is implemented by descending a very high-dimensional loss function. Understanding under which conditions neural networks do not get stuck in poor minima of the loss, and how the landscape of that loss evolves as depth is increased remains a challenge. Here we predict, and test empirically, an analogy between this landscape and the energy landscape of repulsive ellipses. We argue that in fully-connected deep networks a phase transition delimits the over-and under-parametrized regimes where fitting can or cannot be achieved. In the vicinity of this transition, properties of the curvature of the minima of the loss (the spectrum of the hessian) are critical. This transition shares direct similarities with the jamming transition by which particles form a disordered solid as the density is increased, which also occurs in certain classes of computational optimization and learning problems such as the perceptron. Our analysis gives a simple explanation as to why poor minima of the loss cannot be encountered in the overparametrized regime. Interestingly, we observe that the ability of fully-connected networks to fit random data is independent of their depth, an independence that appears to also hold for real data. We also study a quantity ∆ which characterizes how well (∆ < 0) or badly (∆ > 0) a datum is learned. At the critical point it is power-law distributed on several decades, P+(∆) ∼ ∆ θ for ∆ > 0 and P−(∆) ∼ (−∆) −γ for ∆ < 0, with exponents that depend on the choice of activation function. This observation suggests that near the transition the loss landscape has a hierarchical structure and that the learning dynamics is prone to avalanche-like dynamics, with abrupt changes in the set of patterns that are learned.

show abstract

The Generalization Error of Random Features Regression: Precise Asymptotics and the Double Descent Curve

Song

Montanari

2021

Comm Pure Appl Math

121

116

View full text Add to dashboard Cite

Deep learning methods operate in regimes that defy the traditional statistical mindset. Neural network architectures often contain more parameters than training samples, and are so rich that they can interpolate the observed labels, even if the latter are replaced by pure noise. Despite their huge complexity, the same architectures achieve small generalization error on real data.This phenomenon has been rationalized in terms of a so-called 'double descent' curve. As the model complexity increases, the test error follows the usual U-shaped curve at the beginning, first decreasing and then peaking around the interpolation threshold (when the model achieves vanishing training error). However, it descends again as model complexity exceeds this threshold. The global minimum of the test error is found above the interpolation threshold, often in the extreme overparametrization regime in which the number of parameters is much larger than the number of samples. Far from being a peculiar property of deep neural networks, elements of this behavior have been demonstrated in much simpler settings, including linear regression with random covariates.In this paper we consider the problem of learning an unknown function over the d -dimensional sphere S d 1 , from n i.i.d. samples .x i ; y i / P S d 1 ¢ R, i n. We perform ridge regression on N random features of the form .w T a x/, a N . This can be equivalently described as a two-layer neural network with random first-layer weights. We compute the precise asymptotics of the test error, in the limit N; n; d 3 I with N=d and n=d fixed. This provides the first analytically tractable model that captures all the features of the double descent phenomenon without assuming ad hoc misspecification structures. In particular, above a critical value of the signal-to-noise ratio, minimum test error is achieved by extremely overparametrized interpolators, i.e., networks that have a number of parameters much larger than the sample size, and vanishing training error.

show abstract

Scaling description of generalization with number of parameters in deep learning

Cited by 108 publications

References 57 publications

A jamming transition from under- to over-parametrization affects generalization in deep learning

A jamming transition from under- to over-parametrization affects generalization in deep learning

Jamming transition as a paradigm to understand the loss landscape of deep neural networks

The Generalization Error of Random Features Regression: Precise Asymptotics and the Double Descent Curve

Contact Info

Product

Resources

About