Prevalence of neural collapse during the terminal phase of deep learning training

Abstract

Modern practice for training classification deepnets involves a terminal phase of training (TPT), which begins at the epoch where training error first vanishes. During TPT, the training error stays effectively zero, while training loss is pushed toward zero. Direct measurements of TPT, for three prototypical deepnet architectures and across seven canonical classification datasets, expose a pervasive inductive bias we call neural collapse (NC), involving four deeply interconnected phenomena. (NC1) Cross-example within-class variability of last-layer training activations collapses to zero, as the individual activations themselves collapse to their class means. (NC2) The class means collapse to the vertices of a simplex equiangular tight frame (ETF). (NC3) Up to rescaling, the last-layer classifiers collapse to the class means or in other words, to the simplex ETF (i.e., to a self-dual configuration). (NC4) For a given activation, the classifier’s decision collapses to simply choosing whichever class has the closest train class mean (i.e., the nearest class center [NCC] decision rule). The symmetric and very simple geometry induced by the TPT confers important benefits, including better generalization performance, better robustness, and better interpretability.

Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training

“…In this spirit, the aforementioned departure is not as significant as it appears at first glance since the training error is generally, if not always, not exactly zero (see, e.g., ref. (4)). From an optimization point of view, a careful examination indicates that Minority Collapse can be attributed to the two constraints in the Layer-Peeled Model or the 2 regularization in Eq.…”

“…The work of neural collapse by ref. ( 4 ) in this body of work is particularly noticeable with its mathematically elegant and convincing insights. In brief, ref.…”

“…1 , the essence of the modeling strategy is to break down the neural network from top to bottom, specifically singling out the topmost layer and modeling all bottom layers collectively as a single variable. In fact, the top-down perspective that we took in the development of the Layer-Peeled Model was inspired by a recent breakthrough made by Papyan, Han, and Donoho ( 4 ), who discovered a mathematically elegant and pervasive phenomenon termed neural collapse in deep-learning training. This top-down approach was also taken in refs.…”

“…Roughly speaking, neural collapse ( 4 ) refers to the emergence of certain geometric patterns of the last-layer features and the last-layer classifiers , when the neural network for balanced classification problems is well-trained in the sense that it is toward not only zero misclassification error, but also negligible † cross-entropy loss. Specifically, the authors observed the following properties in their massive experiments: The last-layer features from the same class tend to be very close to their class mean; these K -class means centered at the global mean have the same length and form the maximally possible equal-sized angles between any pair; moreover, the last-layer classifiers become dual to the class means in the sense that they are equal to each other for each class up to a scaling factor.…”

“…In brief, ref. ( 4 ) observed the following four properties of the last-layer features and classifiers in deep-learning training on balanced datasets: ‡…”