Space of Functions Computed by Deep-Layered Machines

Mozeika, Alexander; Li, Bo; Saad, David

doi:10.1103/physrevlett.125.168301

Cited by 5 publications

(9 citation statements)

References 29 publications

(38 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Nonetheless, the structure of the mean-field equations can give rise to the same Gaussian processes kernel in the limit of infinite width for both DNNs and RNNs if the readout in the RNN is taken from a single time step. This finding holds for single inputs, as pointed out in [40], as well as input sequences. Furthermore, for a point-symmetric activation function [40], there is no observable difference between DNNs and RNNs on the mean-field level if the biases are uncorrelated in time and the input is only supplied in the first time step.…”

Section: Introductionsupporting

confidence: 69%

“…This finding holds for single inputs, as pointed out in [40], as well as input sequences. Furthermore, for a point-symmetric activation function [40], there is no observable difference between DNNs and RNNs on the mean-field level if the biases are uncorrelated in time and the input is only supplied in the first time step. Going beyond the leading order, we compute the next-to-leadingorder corrections for both DNNs and RNNs.…”

Section: Introductionsupporting

confidence: 69%

“…These methods have been developed in the field of disordered systems, which are systems with random parameters, such as spin glasses [37][38][39], and are able to extract the typical behavior of a system with a large number of interacting components. For example, this approach has recently been used to characterize the typical richness, represented by the entropy, of Boolean functions computed in the output layer of DNNs, RNNs, and sparse Boolean circuits [40].…”

Section: Introductionmentioning

confidence: 99%

“…Concretely, in this paper, we present a systematic derivation of the mean-field theories for DNNs and RNNs that is based on the well-established approach of field theory for recurrent networks [31,35,41,42], which allows a unified treatment of the two architectures [40]. This paves the way for extensions to finite n, n , enabled by a rich set of systematic methods available in the mathematical physics literature to compute corrections beyond the leading order [43,44].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Unified field theoretical approach to deep and recurrent neuronal networks

et al. 2022

View full text Add to dashboard Cite

Understanding capabilities and limitations of different network architectures is of fundamental importance to machine learning. Bayesian inference on Gaussian processes has proven to be a viable approach for studying recurrent and deep networks in the limit of infinite layer width, n → ∞. Here we present a unified and systematic derivation of the mean-field theory for both architectures that starts from first principles by employing established methods from statistical physics of disordered systems. The theory elucidates that while the mean-field equations are different with regard to their temporal structure, they yet yield identical Gaussian kernels when readouts are taken at a single time point or layer, respectively. Bayesian inference applied to classification then predicts identical performance and capabilities for the two architectures. Numerically, we find that convergence towards the mean-field theory is typically slower for recurrent networks than for deep networks and the convergence speed depends non-trivially on the parameters of the weight prior as well as the depth or number of time steps, respectively. Our method exposes that Gaussian processes are but the lowest order of a systematic expansion in 1/n and we compute next-to-leading-order corrections which turn out to be architecture-specific. The formalism thus paves the way to investigate the fundamental differences between recurrent and deep architectures at finite widths n.

show abstract

Section: Introductionsupporting

confidence: 69%

Section: Introductionsupporting

confidence: 69%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Unified field theoretical approach to deep and recurrent neuronal networks

et al. 2022

View full text Add to dashboard Cite

show abstract

“…This could give theoretical backing to computational insights into the robustness and evolvability of circuits [29]. When the number of composition levels is large, this could shed light on the space of functions in some types of neural networks [30].…”

Section: Number Of Permitted Logicsmentioning

confidence: 99%

Boolean composition restricts biological logics

Fink¹,

Hannam²

2021

Preprint

View full text Add to dashboard Cite

Networks of gene regulation determine cell identity and regulate cell function, but little is known about which logics are biologically favored. We show that, remarkably, the number of logical dependencies that a gene can have on others is severely restricted. This is because genes interact via transcription factors but only gene-gene interactions are observed. We enumerate the number of biologically permitted logics by mapping the problem onto the composition of Boolean functions, and confirm our predictions computationally. This is a key insight into how information is processed at the genetic level.

show abstract