Stochastic Normalizations as Bayesian Learning

Abstract

In this work we investigate the reasons why Batch Normalization (BN) improves the generalization performance of deep networks. We argue that one major reason, distinguishing it from data-independent normalization methods, is randomness of batch statistics. This randomness appears in the parameters rather than in activations and admits an interpretation as a practical Bayesian learning. We apply this idea to other (deterministic) normalization techniques that are oblivious to the batch size. We show that their generalization performance can be improved significantly by Bayesian learning of the same form. We obtain test performance comparable to BN and, at the same time, better validation losses suitable for subsequent output uncertainty estimation through approximate Bayesian posterior.Recent advances in hardware and deep NNs make it possible to use large capacity networks, so that the training accuracy becomes close to 100% even for rather difficult tasks. At the same time, however, we would like to ensure small generalization gaps, i.e. a high validation accuracy and a reliable confidence prediction. For this reason, regularization methods become very important.As the base model for this study we have chosen the All-CNN network of [23], a network with eight convolutional layers, and train it on the CIFAR-10 dataset. Recent work [7] compares different regularization techniques with this network and reports test accuracy of 91.87% with their probabilistic network and 90.88% with dropout but omits BN. Fig. 1 shows how well BN generalizes for this problem when applied to exactly the same network. It easily achieves validation accuracy 93%, being significantly better than the dedicated regularization techniques proposed in [7]. It appears that BN is a very powerful regularization method. The goal of this work is to try to understand and exploit the respective mechanism. Towards this end we identify two components: one is a non-linear reparametrization of the model that preconditions gradient descent and the other is stochasticity.The reparametrization may be as well achieved by other normalization techniques such as weight normalization [19] and analytic normalization [22] amongst others [14,1]. The advantage of these methods is that they are deterministic and thus do not rely on batch statistics, often require less computation overhead, are continuously differentiable [22] and can be applied more flexibly, e.g. to cases with a small batch size or recurrent neural networks. Unfortunately, these methods, while improving on the training loss, do not generalize as good as BN, which was observed experimentally in [8,22]. We therefore look at further aspects of BN that could explain its regularization.

Group Whitening: Balancing Learning Efficiency and Representational Capacity

“…We further investigate how batch size affects the training performance of batch normalized networks (Figure 1), from the perspective of a model's representational capacity. Several works [41,18,17] have shown that batch size is related to the magnitude of stochasticity [2,46] introduced by BN, which also affects the model's training performance. However, the stochasticity analysis [18] is specific to normalization along the batch dimension, and cannot explain why GN with a large group number has significantly worse performance (Figure 2), while our work provides a unified analysis for batch and group normalized networks.…”

“…BN standardizes the activations within a mini-batch of data, which improves the conditioning of optimization and accelerates training [20,3,40]. The stochasticity of normalization introduced along the batch dimension is believed to benefit generalization [51,41,18]. However, this stochasticity also results in differences between the training distribution (using mini-batch statistics) and the test distribution (using estimated population statistics) [19], which is believed to be the main cause of BN's smallbatch-size problem -BN's error increases rapidly as the batch size becomes smaller [51].…”

A review of uncertainty quantification in deep learning: Techniques, applications and challenges

“…The proposed DJ approach is flexible procedure which is usable to a wide range of DL methods. Shekhovtsov et al [390] exploited the cause of enhanced the generalization performance of deep networks due to Batch Normalization (BN). They argued that randomness of batch statistics was one of the prime reasons.…”

Normalization Techniques in Training DNNs: Methodology, Analysis and Application

“…One important property of BN is its ability to improve the generalization of DNNs. It is believed such an improvement is obtained from the stochasticity/noise introduced by normalization over batch data [8], [105], [205]. It is clear that both the normalized output (Eqn.17) and the population statistics (Eqn.18) can be viewed as stochastic variables, because they depend on the minibatch inputs, which are sampled over datasets.…”