On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decay

Abstract

Despite the conventional wisdom that using batch normalization with weight decay may improve neural network training, some recent works show their joint usage may cause instabilities at the late stages of training. Other works, in contrast, show convergence to the equilibrium, i.e., the stabilization of training metrics. In this paper, we study this contradiction and show that instead of converging to a stable equilibrium, the training dynamics converge to consistent periodic behavior. That is, the training process regularly exhibits instabilities which, however, do not lead to complete training failure, but cause a new period of training. We rigorously investigate the mechanism underlying this discovered periodic behavior both from an empirical and theoretical point of view and show that this periodic behavior is indeed caused by the interaction between batch normalization and weight decay.

Neural Network Weights Do Not Converge to Stationary Points: An Invariant Measure Perspective

“…However, we should be careful that these analysis does not apply to larger models that do not overfit the data. Second, this shows that the SDE modeling in [Li et al, 2020, Lobacheva et al, 2021 can also be valid. It also shows that our work studies a problem of a different nature (non-zero grad norm).…”

“…These observations motivate us to rethink the convergence proofs used in classical optimization analysis. In addition, a few very recent results reported simialrly large oscillations in Cifar10 training [Li et al, 2020, Kunin et al, 2021, Lobacheva et al, 2021, though the authors focus on SDE approximation or batch normalization. Our work instead focuses on the connection to nonconvex optimization theorems.…”

“…On the right plot, we show the last a few iterations of transformerXL training (see section 6 for more details) and observe even larger oscillations. Furthermore, even in Cifar10 experiments, both Li et al [2020], Lobacheva et al [2021] show very strong periodic divergence in training loss when the number of training epoch is huge (> 1000 epochs). Therefore, we see that the loss looks smooth because the oscillation is much smaller than the longterm loss decrease due to optimization.…”