Computational capabilities of restricted two-layered perceptrons

Priel, Avner; Blatt, Marcelo; Grossmann, Tal; Domany, Eytan; Kanter, Ido

doi:10.1103/physreve.50.577

Cited by 20 publications

(28 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This appears to be in clear contradiction to the results of simulations reported by Barkai et al (1992), Engel et al (1992) and Priel et al (1994) where for K = 3 values of α c between 2 and 2.75 were obtained. Up to 6000 learning cycles were used, a value much larger than what would be expected from Eq.…”

Section: The Committee Machine (Tree Structure)contrasting

confidence: 55%

“…For different sets of patterns similar curves are obtained but we have not tried to average over different sets or to investigate finite size effects. Judging from the simulations performed by Priel et al (1994) N = 150 seems to be sufficient to eliminate drastic finite size effects. There appears to be a qualitative difference between the learning curves for α < α p = 1.75 and α > α p , respectively.…”

Section: The Committee Machine (Tree Structure)mentioning

confidence: 99%

“…In these investigations successful learning of all patterns with probability 1/2 was used as criterion for α c . Priel et al (1994) also determined the median of the time required for perfect learning as function of α. They found for instance for K = 3 and α = 2 the value τ med ≈ 25 and a divergence around α c ≈ 2.75.…”

Section: The Committee Machine (Tree Structure)mentioning

confidence: 99%

“…The question of maximal storage capacity and learning has also been addressed for more complicated networks, for instance a perceptron with binary weights W i = ±1 (Kraut and Mezard (1989), Horner (1992), Horner (1993) and for perceptrons with hidden layers (Barkai et al (1992), Engel et al (1992), Priel et al (1994)). For those architectures α p < α c is expected.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Bounds on learning in polynomial time

Horner

Bethge

1998

Philosophical Magazine B

View full text Add to dashboard Cite

The performance of large neural networks can be judged not only by their storage capacity but also by the time required for learning. A polynomial learning algorithm with learning time ∼ N 2 in a network with N units might be practical whereas a learning time ∼ e N would allow rather small networks only. The question of absolute storage capacity α c and capacity for polynomial learning rules α p is discussed for several feed-forward architectures, the perceptron, the binary perceptron, the committee machine and a perceptron with fixed weights in the first layer and adaptive weights in the second layer. The analysis is based partially on dynamic mean field theory which is valid for N → ∞. Especially for the committee machine a value α p considerably lower than the capacity predicted by replica theory or simulations is found. This discrepancy is resolved by new simulations investigating the learning time dependence and revealing subtleties in the definition of the capacity.

show abstract

Section: The Committee Machine (Tree Structure)contrasting

confidence: 55%

Section: The Committee Machine (Tree Structure)mentioning

confidence: 99%

Section: The Committee Machine (Tree Structure)mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Bounds on learning in polynomial time

Horner

Bethge

1998

Philosophical Magazine B

View full text Add to dashboard Cite

show abstract

“…Specifically, results for a simple perceptron beyond its storage capacity have been utilized for the exploration of multi-layer networks in [201,209].…”

Section: G Multi-layer Perceptronsmentioning

confidence: 99%

Techniques of replica symmetry breaking and the storage problem of the McCulloch–Pitts neuron

Györgyi

2001

Physics Reports

View full text Add to dashboard Cite

In this article we review the framework for spontaneous replica symmetry breaking. Subsequently that is applied to the example of the statistical mechanical description of the storage properties of a McCulloch-Pitts neuron, i. e., simple perceptron. It is shown that in the neuron problem the general formula appears that is at the core of all problems admitting Parisi's replica symmetry breaking ansatz with a one-component order parameter. The details of Parisi's method are reviewed extensively, with regard to the wide range of systems where the method may be applied. Parisi's partial differential equation and related differential equations are discussed, and a Green function technique introduced for the calculation of replica averages, the key to determining the averages of physical quantities. The Green functions of the Fokker-Planck equation due to Sompolinsky turns out to play the role of the statistical mechanical Green function in the graph rules for replica correlators. The subsequently obtained graph rules involve only tree graphs, as appropriate for a mean-field-like model. The lowest order Ward-Takahashi identity is recovered analytically and shown to lead to the Goldstone modes in continuous replica symmetry breaking phases. The need for a replica symmetry breaking theory in the storage problem of the neuron has arisen due to the thermodynamical instability of formerly given solutions. Variational forms for the neuron's free energy are derived in terms of the order parameter function x(q), for different prior distribution of synapses. Analytically in the high temperature limit and numerically in generic cases various phases are identified, among them one similar to the Parisi phase in long range interaction spin glasses. Extensive quantities like the error per pattern change slightly with respect to the known unstable solutions, but there is a significant difference in the distribution of non-extensive quantities like the synaptic overlaps and the pattern storage stability parameter. A simulation result is also reviewed and compared to the prediction of the theory. 07.05. Mh, 75.10Nr, 84.35.+i

show abstract

Efficient dendritic learning as an alternative to synaptic plasticity hypothesis

Hodassman

Vardi

Tugendhaft

et al. 2022

Sci Rep

View full text Add to dashboard Cite

Synaptic plasticity is a long-lasting core hypothesis of brain learning that suggests local adaptation between two connecting neurons and forms the foundation of machine learning. The main complexity of synaptic plasticity is that synapses and dendrites connect neurons in series and existing experiments cannot pinpoint the significant imprinted adaptation location. We showed efficient backpropagation and Hebbian learning on dendritic trees, inspired by experimental-based evidence, for sub-dendritic adaptation and its nonlinear amplification. It has proven to achieve success rates approaching unity for handwritten digits recognition, indicating realization of deep learning even by a single dendrite or neuron. Additionally, dendritic amplification practically generates an exponential number of input crosses, higher-order interactions, with the number of inputs, which enhance success rates. However, direct implementation of a large number of the cross weights and their exhaustive manipulation independently is beyond existing and anticipated computational power. Hence, a new type of nonlinear adaptive dendritic hardware for imitating dendritic learning and estimating the computational capability of the brain must be built.

show abstract

Computational capabilities of restricted two-layered perceptrons

Cited by 20 publications

References 18 publications

Bounds on learning in polynomial time

Bounds on learning in polynomial time

Techniques of replica symmetry breaking and the storage problem of the McCulloch–Pitts neuron

Efficient dendritic learning as an alternative to synaptic plasticity hypothesis

Contact Info

Product

Resources

About