Random Fully Connected Neural Networks as Perturbatively Solvable Hierarchies

Abstract

This article considers fully connected neural networks with Gaussian random weights and biases and L hidden layers, each of width proportional to a large parameter n. For polynomially bounded non-linearities we give sharp estimates in powers of 1/n for the joint correlation functions of the network output and its derivatives. Moreover, we obtain exact layerwise recursions for these correlation functions and solve a number of special cases for classes of non-linearities including ReLU and tanh. We find in both cases that the depth-to-width ratio L/n plays the role of an effective network depth, controlling both the scale of fluctuations at individual neurons and the size of inter-neuron correlations. We use this to study a somewhat simplified version of the so-called exploding and vanishing gradient problem, proving that this particular variant occurs if and only if L/n is large. Several of the key ideas in this article were first developed at a physics level of rigor in a recent monograph [71] with Roberts and Yaida.

Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networks

“…Corrections to this lazy limit can be extracted at small but finite γ 0 . This is conceptually similar to recent works which consider perturbation series for the NTK in powers of 1/N [33,26,27] (though not identical, see Appendix N.7). We expand all observables q(γ 0 ) in a power series in γ 0 , giving q(γ 0 ) = q (0) + γ 0 q (1) + γ 2 0 q (2) + ... and compute corrections up to O(γ 2 0 ).…”

“…, which is consistent with finite width effective field theory at γ = O N (1) [26,27] (Appendix N.6). Further, at the leading order correction, all temporal dependencies are controlled by P (P + 1) functions v α (t) = t 0 ds∆ 0 α (s) and v αβ (t) = t 0 ds∆ 0 α (s) s 0 ds ∆ 0 β (s ), which is consistent with those derived for finite width NNs using a truncation of the Neural Tangent Hierarchy [32,33,26].…”

“…A natural extension to the lazy NTK/NNGP limit that allows the study of feature learning is to calculate finite width corrections to the infinite width limit. Finite width corrections to Bayesian inference in wide networks have been obtained with various perturbative [23][24][25][26][27] and self-consistent techniques [28][29][30][31]. In the gradient descent based setting, leading order corrections to the NTK dynamics have been analyzed to study finite width effects [32][33][34]26].…”

“…We note that unlike other works on perturbation theory in wide networks, we do not attempt to characterize fluctuation effects in the kernels due to finite width, but rather operate in a regime where the kernels are concentrating and their variance is negligible. For a more discussion of perturbative field theory in finite width networks, see [27,26,33].…”

The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initialization

“…In this setting, both the network depth d and the width n of each layer are simultaneously scaled to infinity, while their relative ratio d/n remains fixed [23,[25][26][27][28][29]. Recent work also explores using d/n as an effective perturbation parameter [30][31][32] or to study concentration bounds in terms of d/n [5,33]. This limit has the distinct advantage of being incredibly accurate at predicting the output distribution for finite size networks at initialization [27] -a significant improvement over the NNGP theory.…”