Towards Stabilizing Batch Statistics in Backward Propagation of Batch Normalization

Abstract

Batch Normalization (BN) is one of the most widely used techniques in Deep Learning field. But its performance can awfully degrade with insufficient batch size. This weakness limits the usage of BN on many computer vision tasks like detection or segmentation, where batch size is usually small due to the constraint of memory consumption. Therefore many modified normalization techniques have been proposed, which either fail to restore the performance of BN completely, or have to introduce additional nonlinear operations in inference procedure and increase huge consumption. In this paper, we reveal that there are two extra batch statistics involved in backward propagation of BN, on which has never been well discussed before. The extra batch statistics associated with gradients also can severely affect the training of deep neural network. Based on our analysis, we propose a novel normalization method, named Moving Average Batch Normalization (MABN). MABN can completely restore the performance of vanilla BN in small batch cases, without introducing any additional nonlinear operations in inference procedure. We prove the benefits of MABN by both theoretical analysis and experiments. Our experiments demonstrate the effectiveness of MABN in multiple computer vision tasks including ImageNet and COCO. The code has been released in https://github.com/megvii-model/MABN. * Equal Contribution. Work was done when Junjie Yan was an intern at Megvii Technology. † Corresponding author. 1 In the context of this paper, we use "batch size/normalization batch size" to refer the number of samples used to compute statistics unless otherwise stated. We use "gradient batch size" to refer the number of samples used to update weights.

On the Pitfalls of Batch Normalization for End-to-End Video Learning: A Study on Surgical Workflow Analysis

“…There have been several works investigating the problems of BN [7,35,55,56]. BatchRenorm [23] and MABN [57] are extensions of BatchNorm which aim at reducing the dependence on batches and therefore the train-test discrepancy. Multiple alternative normalization layers have been proposed which only operate on single samples [3,32,33,43,46,52,55], like GroupNorm [55] or LayerNorm [3].…”

Momentum^2 Teacher: Momentum Teacher with Momentum Statistics for Self-Supervised Learning

“…Batch Renormalization [26] and EvalNorm [43] correct batch statistics during training and inference, while compared to Synced BN they generally perform worse. Moving averaged batch normalization [56] and online normalization [11] adopt similar momentum updating of statistics like our momentum BN during the forward pass. However, they need further correct backpropagation for valid SGD optimization, which requires additional computation and memory resources.…”

Exploiting Invariance in Training Deep Neural Networks

“…We also add a small diagonal matrix I to the covariance matrix to avoid rank-deficiency. Note that an attractive idea is to include the population statistics, i.e., the running average of the covariance, for stabilization [44,40]. We have tried but do not pursue this option due to an inherent limitation that the smoothing factor needs to be small to avoid transforming the current batch with a mismatched geometry, leading to a training explosion.…”

“…The wall time can be significantly reduced if we increase the batch size and reduce the number of function calls. At batch size 64, our slowest run takes 11 hours (3 hours faster), reaching Box AP 38.7, Mask AP 35.6, superior to a recent method that trains with 2× more steps[40]. Besides optimizing these operations through low-level changes to the software package, we can also use standardization layers in only a subset of the network.…”