A jamming transition from under- to over-parametrization affects generalization in deep learning

Abstract

In this paper we first recall the recent result that in deep networks a phase transition, analogous to the jamming transition of granular media, delimits the over-and under-parametrized regimes where fitting can or cannot be achieved. The analysis leading to this result support that for proper initialization and architectures, in the whole over-parametrized regime poor minima of the loss are not encountered during training, because the number of constraints that hinders the dynamics is insufficient to allow for the emergence of stable minima. Next, we study systematically how this transition affects generalization properties of the network (i.e. its predictive power). As we increase the number of parameters of a given model, starting from an under-parametrized network, we observe for gradient descent that the generalization error displays three phases: (i) initial decay, (ii) increase until the transition point -where it displays a cusp -and (iii) slow decay toward an asymptote as the network width diverges. However if early stopping is used, the cusp signaling the jamming transition disappears. Thereby we identify the region where the classical phenomenon of over-fitting takes place as the vicinity of the jamming transition, and the region where the model keeps improving with increasing the number of parameters, thus organizing previous empirical observations made in modern neural networks. ‡ The often used cross-entropy loss function also displays a transition where all data are well-fitted. However, in the over-parametrized regime the dynamic never stops, as the total loss vanishes only if the output and therefore the weights diverge. Imposing a time cut-off is done in practice, but it blurs the criticality near jamming, as exemplified below with the early stopping procedure.

Scaling description of generalization with number of parameters in deep learning

“…Recent works suggest that the two questions above are closely connected. Numerical and theoretical studies [6,7,8,9,10,11,12,13,14,15,16,17] show that in the overparametrized regime, the loss landscape of DNNs is not rough with isolated minima as initially thought [18,19], but instead has connected level sets and presents many flat directions, even near its global minimum. In particular, recent works on the over-parametrized regime of DNNs [20,21,22,23] have shown that the landscape around a typical initialization point becomes essentially convex, allowing for convergence to a global minimum during training.…”

“…In [16,17], it has been observed that when optimizing DNNs (using the so-called hinge loss), there is a sharp phase transition -whose location can depend on the chosen dynamics -at some N * (P ) such that for N ≥ N * the dynamic process reaches a global minimum of the loss. In particular whenever N > N * , the training error (i.e.…”

“…the probability that a data point outside of the training set is correctly classified) [24,25,26,27]. Indeed the test error (the probability of an incorrect classification for an unseen data point) has been observed to decrease as N → ∞ in a slow power-law fashion [17]. In contrast, as N → N * , the test error blows up [27,28,17] (a phenomenon shown by the blue curve in Fig.…”

Heavy-Tailed Universality Predicts Trends in Test Accuracies for Very Large Pre-Trained Deep Neural Networks

“…HT-MU applies to the analysis of complicated systems, including many physical systems, traditional NNs [23,24], and even models of the dynamics of actual spiking neurons. Indeed, the dynamics of learning in DNNs seems to resemble a system near a phase transition, such as the phase boundary of spin glass, or a system displaying Self Organized Criticality (SOC), or a Jamming transition [25,26]. Of course, we can not say which mechanism, if any, is at play.…”

Making Learners (More) Monotone

“…For the peaking phenomema [2], this occurs exactly at the transition from the underparametrized to the overparametrized regime. This phenomena has regained interest in the machine learning community in the context of deep neural networks [5,6], since these models are typically overparametrized. Recently, also several new examples have been found, where in quite simple settings more data results in worse generalization performance [7,8].…”