Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Function

Abstract

We demonstrate that in residual neural networks (ResNets) dynamical isometry is achievable irrespective of the activation function used. We do that by deriving, with the help of Free Probability and Random Matrix Theories, a universal formula for the spectral density of the input-output Jacobian at initialization, in the large network width and depth limit. The resulting singular value spectrum depends on a single parameter, which we calculate for a variety of popular activation functions, by analyzing the signal propagation in the artificial neural network. We corroborate our results with numerical simulations of both random matrices and ResNets applied to the CIFAR-10 classification problem. Moreover, we study consequences of this universal behavior for the initial and late phases of the learning processes. We conclude by drawing attention to the simple fact, that initialization acts as a confounding factor between the choice of activation function and the rate of learning. We propose that in ResNets this can be resolved based on our results by ensuring the same level of dynamical isometry at initialization. I. INTRODUCTIONDeep Learning has achieved unparalleled success in fields such as object detection and recognition, language translation, and speech recognition [16]. At the same time, models achieving these state-of-the-art results are increasingly deep and complex [3], which often leads to optimization challenges such as vanishing gradients. Many solutions to this problem have been proposed. In particular, Residual Neural Networks remedy this to some extent [10,35] by using skip connections in the network architecture, which improve gradient flow. As a result, Residual Neural Networks outmatched other competing models in the 2015 ILSVRC and COCO competitions. Yet another approach towards solving this problem is to tailor fit the networks weight initialization to facilitate training, for example by ensuring dynamical isometry [28]. In this latter case, the insights are based on an analysis of the statistical properties of information propagation in the network and a study of the full singular spectrum of a particular matrix, namely the input-output Jacobian, via the techniques of Free Probability and Random Matrix Theories (FPT & RMT). This perspective has recently led to successfully training a 10000 layer vanilla convolutional neural network [38].RMT is a versatile tool that, since its inception, saw a substantial share of applications, from the earliest in nuclear physics [37] to the latest in game theory [4] (see [1] for some of the use cases discovered in the mean time). It is thus not surprising that it found its way to be used to understand artificial neural networks. In particular, to study their loss surface [6,27], the associated Gram matrix [19,30] and in the case of single layer networks, their dynamics [17]. Our main contribution is extending the theoretical analysis of [28,29,33] to residual networks. In particular, we find that residual networks can achieve dynamical isometry for many differe...

Residual Networks as Nonlinear Systems: Stability Analysis using Linearization

“…stable. In a similar spirit, [14,15] apply random matrix theory and mean field theory to investigate the Jacobian in the limit of extreme deep and/or wide networks. These papers focus on the behavior of the Jacobian of (usually simplified) networks during training, whereas our objective is to understand the behavior of pre-trained networks as they are used in practice, and to this end we also primarily consider the Jacobians of the network components rather than the network as a whole.…”

A Comprehensive and Modularized Statistical Framework for Gradient Norm Equality in Deep Neural Networks

“…A neural network is dynamical isometry as long as every singular value of its input-output Jacobian matrix remains close to 1, thus the norm of every error vector and the angle between error vectors are preserved. With the powerful theorems of free probability and random matrix, Pennington et al (2017) [15] investigate the spectrum density distribution of plaint fully-connected serial network with Gaussian/orthogonal weights and ReLU/hard-tanh activation functions; Tarnowski et al (2018) [16] explore the density of singular values of the input-output Jacobian matrix in ResNet and identify that dynamical isometry can be always achieved regardless of the choice of activation function. However, their studies only cover ResNet whose major branch consists of Gaussian and scaled orthogonal linear transforms and activation functions, and fail to provide a theoretical explanation of batch normalization.…”

“…If the network seats steady on the border between the order and chaos phase, it will be trainable even with a depth of 10,000 [4]. Pennington et al (2017) [15], Tarnowski et al (2018) [16], [17] and Ling & Qiu (2018) [18] argue that networks achieving dynamical isometry (all the singular value of the network's input-output Jacobian matrix remain close to 1) do not suffer from gradient explosion or vanishing. Philipp et al (2019) [19] directly evaluate the statistics of the gradient and propose a metric called gradient scale coefficient (GSC) that can verify whether a network would suffer gradient explosion.…”

“…For example, to calculate the quadratic mean norm (qm norm) of the Jacobian matrices, Philipp et al (2019) [19] assume that the norm of the product of Jacobian matrices has approximate decomposability. The free probability used in Pennington et al (2017) [15], Tarnowski et al (2018) [16], [17] and Ling & Qiu (2018) [18] requires the involved matrices to be freely independent with each other [21], which is not commonly held and difficult to verify [22]. Because of the complex derivation, existing studies usually require strong statistics backgrounds and mathematical skills, which constraints their spread in the community.…”

How to Initialize your Network? Robust Initialization for WeightNorm & ResNets

“…Glorot and Bengio [7], He et al [11], Saxe et al [25], Poole et al [23], Pennington et al [21,22], to name some of the most prominent ones). Similarly, previous works have studied initialization strategies for un-normalized ResNets [10,31,32], but they lack large scale experiments demonstrating the effectiveness of the proposed approaches and consider a simplified ResNet setup where shortcut connections are ignored, even though they play an important role [15]. Zhang et al [37] propose an initialization scheme for un-normalized ResNets which involves initializing the different types of layers individually using carefully designed schemes.…”