Dynamics in Deep Classifiers Trained with the Square Loss: Normalization, Low Rank, Neural Collapse, and Generalization Bounds

Abstract

We overview several properties—old and new—of training overparameterized deep networks under the square loss. We first consider a model of the dynamics of gradient flow under the square loss in deep homogeneous rectified linear unit networks. We study the convergence to a solution with the absolute minimum ρ , which is the product of the Frobenius norms of each layer weight matrix, when normalization by Lagrange multipliers is used together with weight decay under different forms of gradient descent. A main property of the minimizers that bound their expected error for a specific network architecture is ρ . In particular, we derive novel norm-based bounds for convolutional layers that are orders of magnitude better than classical bounds for dense networks. Next, we prove that quasi-interpolating solutions obtained by stochastic gradient descent in the presence of weight decay have a bias toward low-rank weight matrices, which should improve generalization. The same analysis predicts the existence of an inherent stochastic gradient descent noise for deep networks. In both cases, we verify our predictions experimentally. We then predict neural collapse and its properties without any specific assumption—unlike other published proofs. Our analysis supports the idea that the advantage of deep networks relative to other classifiers is greater for problems that are appropriate for sparse deep architectures such as convolutional neural networks. The reason is that compositionally sparse target functions can be approximated well by “sparse” deep networks without incurring in the curse of dimensionality.

Efficient Simultaneous Detection of Metabolites Based on Electroenzymatic Assembly Strategy

“…The SPSS software was used for linear regression analysis [ 36 ] of the first group of data (all data corresponding to the test gradient under the plasma background selected from the full set of gradient data tested in the PBS), second group of data (all data tested in the plasma), and third group of data (full gradient data tested in the PBS). The detailed calculation processes are presented in the Supplementary Materials.…”

Self-transformation of encoded subregion image data augmentation for better classification

“…The above methods have to some extent improved the generalization ability of the model, but they all heavily rely on the parameter settings of the higher-level neural network, which has significant limitations. The implicit regularization method reduces the generalization error of the model without limiting the representation ability, such as batch normalization 19 and weight normalization 20 . They accelerate the convergence speed of the model and improve the generalization ability of the model by smoothing the variance between data or by normalizing the parameter vector to unit length, coupling the weight matrix decomposition in the network into modules and directions.…”

“…The implicit regularization method reduces the generalization error of the model without limiting the representation ability, such as batch normalization 19 and weight normalization. 20 They accelerate the convergence speed of the model and improve the generalization ability of the model by smoothing the variance between data or by normalizing the parameter vector to unit length, coupling the weight matrix decomposition in the network into modules and directions. Thus, the translation and rotation invariance of the input data in the model is enhanced.…”

Compositional sparsity of learnable functions

“…In the more interesting overparametrized square loss case, generalization depends on solving a sort of regularized ERM that consists of finding minimizers of the empirical risk with zero loss, and then selecting the one with lowest complexity. Recent work [38] has provided theoretical and empirical evidence that this can be accomplished by SGD provided that the following conditions are satisfied:…”

“…Higher complexity generally gives higher (upper or lower bounds on) generalization error. Theorem 4.1 [8,37] therefore implies that sparsity of a network dramatically reduces generalization error.…”