When and how epochwise double descent happens

Abstract

Deep neural networks are known to exhibit a 'double descent' behavior as the number of parameters increases. Recently, it has also been shown that an 'epochwise double descent' effect exists in which the generalization error initially drops, then rises, and finally drops again with increasing training time. This presents a practical problem in that the amount of time required for training is long, and early stopping based on validation performance may result in suboptimal generalization. In this work we develop an analytically tractable model of epochwise double descent that allows us to characterise theoretically when this effect is likely to occur. This model is based on the hypothesis that the training data contains features that are slow to learn but informative. We then show experimentally that deep neural networks behave similarly to our theoretical model. Our findings indicate that epochwise double descent requires a critical amount of noise to occur, but above a second critical noise level early stopping remains effective. Using insights from theory, we give two methods by which epochwise double descent can be removed: one that removes slow to learn features from the input and reduces generalization performance, and another that instead modifies the training dynamics and matches or exceeds the generalization performance of standard training. Taken together, our results suggest a new picture of how epochwise double descent emerges from the interplay between the dynamics of training and noise in the training data. * Work done at Intel Labs Preprint. Under review.

Multi-scale Feature Learning Dynamics: Insights for Double Descent

“…Our findings and those of Heckel & Yilmaz (2020) and Stephenson & Lee (2021) reinforce one another with a common central finding that the epoch-wise double descent results from different features/layers being learned at different time-scales. However, we also highlight that both Heckel & Yilmaz (2020) and Stephenson & Lee (2021) use tools from random matrix theory to study distinct data models from our teacher-student setup. We study a similar phenomenon by leveraging the replica method from statistical physics to characterize the generalization behavior using a set of informative macroscopic parameters.…”

“…In recent years, there has been an interest in studying the non-asymptotic (finite training time) performance (e.g. Saxe et al, 2013;Advani & Saxe, 2017;Nakkiran et al, 2019b;Pezeshki et al, 2020a;Stephenson & Lee, 2021). Among the limited work studying the particular epoch-wise double descent, Nakkiran et al (2019a) introduces the notion of effective model complexity and hypothesizes that it increases with training time and hence unifies both model-wise and epoch-wise double descent.…”