Multi-scale Feature Learning Dynamics: Insights for Double Descent

Abstract

A key challenge in building theoretical foundations for deep learning is the complex optimization dynamics of neural networks, resulting from the highdimensional interactions between the large number of network parameters. Such non-trivial dynamics lead to intriguing behaviors such as the phenomenon of "double descent" of the generalization error. The more commonly studied aspect of this phenomenon corresponds to model-wise double descent where the test error exhibits a second descent with increasing model complexity, beyond the classical U-shaped error curve. In this work, we investigate the origins of the less studied epoch-wise double descent in which the test error undergoes two non-monotonous transitions, or descents as the training time increases. By leveraging tools from statistical physics, we study a linear teacher-student setup exhibiting epoch-wise double descent similar to that in deep neural networks. In this setting, we derive closed-form analytical expressions for the evolution of generalization error over training. We find that double descent can be attributed to distinct features being learned at different scales: as fast-learning features overfit, slower-learning features start to fit, resulting in a second descent in test error. We validate our findings through numerical experiments where our theory accurately predicts empirical findings and remains consistent with observations in deep neural networks.

Generalizing Similarity in Noisy Setups: The DIBS Phenomenon

“…Limitations. This is an exploratory work that does not investigate all possible setups which may affect or lead to DD, such as regularization (see [37,34]), epoch and sample-wise DD (see, [36,4,22,42]). Moreover, we focus on the under-and overparametrized regime without providing quantitative results about the interpolation threshold itself, [13,14,34].…”

“…Over the past few years, significant strides have been made to understand how neural networks generalize in the presence of noise in classification problems (e.g., [31,19,1,20,47]). Remarkably, the DD phenomenon enabled a closer examination of the NN behavior as the number of trainable parameters, the evolution time, and the size of the dataset vary [36,4,22,42]. Subsequently, other works have produced analytical studies of some of these phenomena…”