Noisy Recurrent Neural Networks

Lim, Soon Hoe; Erichson, N. Benjamin; Hodgkinson, Liam; Mahoney, Michael W.

doi:10.48550/arxiv.2102.04877

Cited by 7 publications

(11 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…which is consistent with the standard convention used in the literature (e.g., [27,45]), in which the weight matrix acts on the output of the activation function, rather than appearing in its argument as in [48]. For example, to recover a multilayer perceptron with preactivations at layer defined as h = W ij φ(h −1 j ) + b , we set γ = 0 and A = B = [0], and view the discrete timesteps as subsequent layers ∈ [0, L − 1].…”

Section: Self-averaging Random Networksupporting

confidence: 80%

See 1 more Smart Citation

The edge of chaos: quantum field theory and deep neural networks

Jefferson¹

2021

Preprint

View full text Add to dashboard Cite

We explicitly construct the quantum field theory corresponding to a general class of deep neural networks encompassing both recurrent and feedforward architectures. We first consider the mean-field theory (MFT) obtained as the leading saddlepoint in the action, and derive the condition for criticality via the largest Lyapunov exponent. We then compute the loop corrections to the correlation function in a perturbative expansion in the ratio of depth T to width N , and find a precise analogy with the well-studied O(N ) vector model, in which the variance of the weight initializations plays the role of the 't Hooft coupling. In particular, we compute both the O(1) corrections quantifying fluctuations from typicality in the ensemble of networks, and the subleading O(T /N ) corrections due to finite-width effects. These provide corrections to the correlation length that controls the depth to which information can propagate through the network, and thereby sets the scale at which such networks are trainable by gradient descent. Our analysis provides a first-principles approach to the rapidly emerging NN-QFT correspondence, and opens several interesting avenues to the study of criticality in deep neural networks.

show abstract

Section: Self-averaging Random Networksupporting

confidence: 80%

“…Our starting point is the continuous-time formulation of a recurrent neural network (RNN) as a stochastic differential equation (SDE) [48] dh = f (h, x) dt + g(h, x) dB ,…”

Section: Constructing the Partition Functionmentioning

confidence: 99%

The edge of chaos: quantum field theory and deep neural networks

Jefferson¹

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Our starting point is the continuous-time formulation of a recurrent neural network (RNN) as a stochastic differential equation (SDE) [51]…”

Section: Constructing the Partition Functionmentioning

confidence: 99%

“…In principle however, one could consider other forms of stochasticity; see [51] and references therein.…”

Section: A1 Simplifying Assumptionsmentioning

confidence: 99%

The edge of chaos: quantum field theory and deep neural networks

Jefferson

2022

SciPost Phys.

View full text Add to dashboard Cite

We explicitly construct the quantum field theory corresponding to a general class of deep neural networks encompassing both recurrent and feedforward architectures. We first consider the mean-field theory (MFT) obtained as the leading saddlepoint in the action, and derive the condition for criticality via the largest Lyapunov exponent. We then compute the loop corrections to the correlation function in a perturbative expansion in the ratio of depth T to width N, and find a precise analogy with the well-studied O(N) vector model, in which the variance of the weight initializations plays the role of the 't Hooft coupling. In particular, we compute both the O(1) corrections quantifying fluctuations from typicality in the ensemble of networks, and the subleading O(T/N) corrections due to finite-width effects. These provide corrections to the correlation length that controls the depth to which information can propagate through the network, and thereby sets the scale at which such networks are trainable by gradient descent. Our analysis provides a first-principles approach to the rapidly emerging NN-QFT correspondence, and opens several interesting avenues to the study of criticality in deep neural networks.

show abstract

“…More generally, the connection between deep learning and dynamical systems has recently motivated several physicsbased models [2,3,17,46,47,57], as well as more stable [11,25,26] and more robust [1,28,44] models.…”

Section: Related Workmentioning

confidence: 99%

Stateful ODE-Nets using Basis Function Expansions

Queiruga¹,

Erichson²,

Hodgkinson³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

The recently-introduced class of ordinary differential equation networks (ODE-Nets) establishes a fruitful connection between deep learning and dynamical systems. In this work, we reconsider formulations of the weights as continuous-depth functions using linear combinations of basis functions. This perspective allows us to compress the weights through a change of basis, without retraining, while maintaining near state-of-the-art performance. In turn, both inference time and the memory footprint are reduced, enabling quick and rigorous adaptation between computational environments. Furthermore, our framework enables meaningful continuous-in-time batch normalization layers using function projections. The performance of basis function compression is demonstrated by applying continuous-depth models to (a) image classification tasks using convolutional units and (b) sentence-tagging tasks using transformer encoder units.

show abstract

Noisy Recurrent Neural Networks

Cited by 7 publications

References 0 publications

The edge of chaos: quantum field theory and deep neural networks

The edge of chaos: quantum field theory and deep neural networks

The edge of chaos: quantum field theory and deep neural networks

Stateful ODE-Nets using Basis Function Expansions

Contact Info

Product

Resources

About