If dropout limits trainable depth, does critical initialisation still matter? A large-scale statistical analysis on ReLU networks

Large deviation analysis of function sensitivity in random deep neural networks

“…We first consider the effect of weight perturbation by sparsifying connections as in equation (15). For a concrete example, we consider DNN with L = 4, uniform layer width α l = 1 and disconnection probability p l = 1/2, for which we compute the large deviation rate function I(q L ) = Φ(Q * , q * , .…”

“…A recent line of research utilizes the mean field theory in statistical physics to investigate various DNN characteristics, such as expressive power [9], Gaussian process-like behaviors of wide DNN [10][11][12], dynamical stability in layer propagation and its impact on weight initialization [13][14][15] and function similarity and entropy in the function space [16]. By assuming large layer-width and random weights, such techniques harness the specific type of nonlinearity used and many degrees of freedom to provide valuable analytical insights.…”

“…We have also found that ReLU activation induces correlations among variables in random convolution networks [16]. The robustness of random networks with ReLU activation is related to the simplicity of the functions they compute [21,22], which may converge to a constant function in the large depth and width limit [15], although, in principle, they admit high capacity with arbitrary weights. However, DNN used in practice are of finite size and finite depth, therefore it is essential to analyze the deviation of finite-size systems with respect to the typical mean field behavior, and characterize its rate of convergence with increasing size.…”