Large deviation analysis of function sensitivity in random deep neural networks

Li, Bo; Saad, David

doi:10.1088/1751-8121/ab6a6f

Cited by 9 publications

(9 citation statements)

References 23 publications

(57 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, it is also highly challenging due to the inherent recursiveness of computation and randomness in their architecture and/or computing elements. Existing theoretical studies of the function space of deep-layered machines are mostly based on the mean field approach, which allows for a sensitivity analysis of the functions realized by deep-layered machines due to input or parameter perturbations [4,[18][19][20].…”

mentioning

confidence: 99%

Space of Functions Computed by Deep-Layered Machines

Mozeika

Saad

2020

Phys. Rev. Lett.

Self Cite

View full text Add to dashboard Cite

We study the space of functions computed by random-layered machines, including deep neural networks and Boolean circuits. Investigating the distribution of Boolean functions computed on the recurrent and layer-dependent architectures, we find that it is the same in both models. Depending on the initial conditions and computing elements used, we characterize the space of functions computed at the large depth limit and show that the macroscopic entropy of Boolean functions is either monotonically increasing or decreasing with the growing depth.

show abstract

mentioning

confidence: 99%

Space of Functions Computed by Deep-Layered Machines

Mozeika

Saad

2020

Phys. Rev. Lett.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Unlike Ref. [40,41], which studies perturbation around a ReLU network, our analysis aims to understand the critical properties of correlations. We consider correlations within the set of weights (w l i ) incoming to each neuron i, with all neurons identically distributed.…”

Section: Mean Field Analysis Of Signal Propagation With Correlated We...mentioning

confidence: 99%

“…A growing body of work has analyzed signal propagation in infinitely wide networks to understand forward-propagation in DNNs [35,36,37,38,39,40,41]. We mention a few results for ReLU networks.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Initializing ReLU networks in an expressive subspace of weights

Singh,

Sreejith

2021

Preprint

View full text Add to dashboard Cite

Using a mean-field theory of signal propagation, we analyze the evolution of correlations between two signals propagating through a ReLU network with correlated weights. Signals become highly correlated in deep ReLU networks with uncorrelated weights. We show that ReLU networks with anti-correlated weights can avoid this fate and have a chaotic phase where the correlations saturate below unity. Consistent with this analysis, we find that networks initialized with anti-correlated weights can train faster (in a teacher-student setting) by taking advantage of the increased expressivity in the chaotic phase. Combining this with a previously proposed strategy of using an asymmetric initialization to reduce dead ReLU probability, we propose an initialization scheme that allows faster training and learning than the best-known methods.

show abstract

“…Xiang et al [46] introduced the maximum sensitivity by a bounded disturbance on the nominal input to measure the maximum deviation of outputs. Li et al [47] studied the deviation of functions represented by DNN from their typical mean field solutions by the large deviation theory and path integral analysis, where the commonly used weight sparsification and binarization in model simplification were investigated under parameter perturbations. In [48], a provable Sensitivity-informed Provable Pruning (SiPPing) method of neural networks was suggested based on a ST-SA of measuring the importance of each weight for one layer.…”

Section: A the Stochastic Sensitivity Analysis Of Neural Networkmentioning

confidence: 99%

Bilateral Sensitivity Analysis for Understandable Neural Networks and its application to Reservoir Engineering

Wang¹,

Zhang²,

Zhang³

et al. 2020

Preprint

View full text Add to dashboard Cite

In this paper, a model-independent sensitivity analysis for (deep) neural network, Bilateral Sensitivity Analysis (BiSA), is proposed to measure the relationship between neurons and layers. Both the BiSA between pair of layers and the BiSA between any pair neurons in different layers are defined for (deep) neural networks. This sensitivity can measure the influence or contribution from any layer to another layer behind this layer in the (deep) neural networks. It provides a helpful tool to interpret the learned model. The BiSA can also measure the influence or contribution from any neuron to another neuron in a subsequent layer and is critical to analyze the relationship between neurons in different layers. Then the BiSA from any input to any output of a network is easily defined to assess the connections between the inputs and outputs. The proposed BiSA of (deep) neural networks is then applied to characterize the well connectivity in reservoir engineering. Given a network trained by Water Injection Rates (WIRs) and Liquid Production Rates (LPRs) data, the well connectivity can be efficiently described through BiSA. The empirical results verify the effectiveness of the proposed method. The comparisons with the exiting methods demonstrate the robustness and the superior performance of the proposed method.

show abstract

Large deviation analysis of function sensitivity in random deep neural networks

Cited by 9 publications

References 23 publications

Space of Functions Computed by Deep-Layered Machines

Space of Functions Computed by Deep-Layered Machines

Initializing ReLU networks in an expressive subspace of weights

Bilateral Sensitivity Analysis for Understandable Neural Networks and its application to Reservoir Engineering

Contact Info

Product

Resources

About