Jamming transition as a paradigm to understand the loss landscape of deep neural networks

Abstract

Deep learning has been immensely successful at a variety of tasks, ranging from classification to artificial intelligence. Learning corresponds to fitting training data, which is implemented by descending a very high-dimensional loss function. Understanding under which conditions neural networks do not get stuck in poor minima of the loss, and how the landscape of that loss evolves as depth is increased remains a challenge. Here we predict, and test empirically, an analogy between this landscape and the energy landscape of repulsive ellipses. We argue that in fully-connected deep networks a phase transition delimits the over-and under-parametrized regimes where fitting can or cannot be achieved. In the vicinity of this transition, properties of the curvature of the minima of the loss (the spectrum of the hessian) are critical. This transition shares direct similarities with the jamming transition by which particles form a disordered solid as the density is increased, which also occurs in certain classes of computational optimization and learning problems such as the perceptron. Our analysis gives a simple explanation as to why poor minima of the loss cannot be encountered in the overparametrized regime. Interestingly, we observe that the ability of fully-connected networks to fit random data is independent of their depth, an independence that appears to also hold for real data. We also study a quantity ∆ which characterizes how well (∆ < 0) or badly (∆ > 0) a datum is learned. At the critical point it is power-law distributed on several decades, P+(∆) ∼ ∆ θ for ∆ > 0 and P−(∆) ∼ (−∆) −γ for ∆ < 0, with exponents that depend on the choice of activation function. This observation suggests that near the transition the loss landscape has a hierarchical structure and that the learning dynamics is prone to avalanche-like dynamics, with abrupt changes in the set of patterns that are learned.

Harmless Interpolation of Noisy Data in Regression

“…(The example 5 in Section 4 is just a caricature in that direction.) In the empirical double descent papers [37,38], the second descent of the test MSE in the overparameterized regime is strictly better than the first descent in the underparameterized regime. From our concrete understanding of the correctly specified high-dimensional regime, it is clear that this is only possible in the presence of an approximation-theoretic benefit of adding more features into the model.…”

“…Most recently, a double-descent curve on the test error (0 − 1 loss and MSE) as a function of the number of parameters of several parametric models was observed on several common datasets by physicists [37] and machine learning researchers [38] respectively. In these experiments, the minimum 2 -norm interpolating solution is used, and several feature families, including kernel approximators [39], were considered.…”

“…Since the publication 8 of the double descent experiments [37,38] at the end of the year 2018, there has been extremely active interest in understanding the generalization abilities of interpolating solutions in linear regression. An earlier edition of our work was presented at Information Theory and Applications, February 2019 and subsequently accepted to IEEE International Symposium on Information Theory, July 2019.…”

Heavy-Tailed Universality Predicts Trends in Test Accuracies for Very Large Pre-Trained Deep Neural Networks

“…HT-MU applies to the analysis of complicated systems, including many physical systems, traditional NNs [23,24], and even models of the dynamics of actual spiking neurons. Indeed, the dynamics of learning in DNNs seems to resemble a system near a phase transition, such as the phase boundary of spin glass, or a system displaying Self Organized Criticality (SOC), or a Jamming transition [25,26]. Of course, we can not say which mechanism, if any, is at play.…”

From complex to simple : hierarchical free-energy landscape renormalized in deep neural networks

“…Understanding the nature of such glass transitions and jamming is a fundamental problem in CSPs since it is intimately related to efficiency of algorithms to solve CSPs. In the context of DNN, it is certainly important to understand the characteristics of the free-energy landscape to understand the efficiently of various learning algorithms for DNNs [18][19][20].…”