A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks

Abstract

The gradient noise (GN) in the stochastic gradient descent (SGD) algorithm is often considered to be Gaussian in the large data regime by assuming that the classical central limit theorem (CLT) kicks in. This assumption is often made for mathematical convenience, since it enables SGD to be analyzed as a stochastic differential equation (SDE) driven by a Brownian motion. We argue that the Gaussianity assumption might fail to hold in deep learning settings and hence render the Brownian motion-based analyses inappropriate. Inspired by non-Gaussian natural phenomena, we consider the GN in a more general context and invoke the generalized CLT (GCLT), which suggests that the GN converges to a heavy-tailed α-stable random variable. Accordingly, we propose to analyze SGD as an SDE driven by a Lévy motion. Such SDEs can incur 'jumps', which force the SDE transition from narrow minima to wider minima, as proven by existing metastability theory. To validate the α-stable assumption, we conduct extensive experiments on common deep learning architectures and show that in all settings, the GN is highly non-Gaussian and admits heavy-tails. We further investigate the tail behavior in varying network architectures and sizes, loss functions, and datasets. Our results open up a different perspective and shed more light on the belief that SGD prefers wide minima.

A Fully Spiking Hybrid Neural Network for Energy-Efficient Object Detection

“…For this case, we consider the unsupervised learning method described in [30]. We perform a tail index analysis of STDP in deep SNN, similar to the method adopted in [31].…”

“…Similar to the analysis done by Simsekli et. al, we evaluate the tail-index of the standard CNN trained with the SGD and the spiking convolutional neural networks trained with STDP processes [31]. The experiment is repeated with varying numbers of layers and is evaluated on the MNIST dataset and the tail indices are reported in Table I.…”

Understanding Short-Range Memory Effects in Deep Neural Networks

“…The first issue relates to the infiniteness of the second moment of SGN, assumed in [26] [27]. According to (11), the noise covariance is the product of the sampling noise and the gradient matrix…”

“…Furthermore, it was recently argued that the critical assumption that SGN follows a Gaussian distribution does not necessarily hold [26] [27]. Instead, it was assumed that the second moment of SGN may not be finite, and it was concluded that the noise follows a Lévy random process by invoking the generalized CLT.…”

Near-Optimal High Probability Complexity Bounds for Non-Smooth Stochastic Optimization with Heavy-Tailed Noise

“…Recent studies [36,35,42] show that in several popular problems such as training BERT [38] on Wikipedia dataset the noise in the stochastic gradients is heavy-tailed. Moreover, in [42], the authors justify empirically that in such cases SGD works significantly worse than clipped-SGD [31] and Adam.…”