Finite Size Scaling in Neural Networks

Nadler, Walter; Fink, Wolfgang

doi:10.1103/physrevlett.78.555

Cited by 6 publications

(5 citation statements)

References 23 publications

(45 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Notice, however, that the crossing point is at some probability P < 1/2. Similar curves, obtained for other architectures, such as the parity and commitee machines [7] crossed at P > 1/2. If there is a sharp transition in the thermodynamic limit (N ∞), these curves should approach a step-function, with…”

Section: Finite Size Scaling Analysissupporting

confidence: 74%

“…To understand the geometrical meaning of this requirement, note that when the concentration of odorant µ is varied in the allowed range (1), the corresponding vector S µ traces a curve (or string) in the N -dimensional space of sensory responses. The requirement (7) means that there exists a hyperplane, such that the entire curve that corresponds to the target odorant lies on one side of it, while the curves that correspond to all p background odorants lie on the other side. This explains our statement, made in the Introduction, that the problem we solve deals with the Linear Separability of curves.…”

Section: A Odorant Identification As Linear Separation Of Curves In N...mentioning

confidence: 99%

“…In order to obtain these results using existing methodology, the most natural and straightforward thing to do is to place a discrete set of ζ = 1, 2, ..., M points S µζ on each curve, corresponding to different concentrations, and to require that the M points that lie on the curve of the target odorant are linearly separable from the P M points that represent the background. That is, equations (7) become…”

Section: A Odorant Identification As Linear Separation Of Curves In N...mentioning

confidence: 99%

See 2 more Smart Citations

Computational Capacity of an Odorant Discriminator: The Linear Separability of Curves

et al. 2002

View full text Add to dashboard Cite

We introduce and study an artificial neural network inspired by the probabilistic receptor affinity distribution model of olfaction. Our system consists of N sensory neurons whose outputs converge on a single processing linear threshold element. The system's aim is to model discrimination of a single target odorant from a large number p of background odorants within a range of odorant concentrations. We show that this is possible provided p does not exceed a critical value p(c) and calculate the critical capacity alpha(c) = p(c)/N. The critical capacity depends on the range of concentrations in which the discrimination is to be accomplished. If the olfactory bulb may be thought of as a collection of such processing elements, each responsible for the discrimination of a single odorant, our study provides a quantitative analysis of the potential computational properties of the olfactory bulb. The mathematical formulation of the problem we consider is one of determining the capacity for linear separability of continuous curves, embedded in a large-dimensional space. This is accomplished here by a numerical study, using a method that signals whether the discrimination task is realizable, together with a finite-size scaling analysis.

show abstract

Section: Finite Size Scaling Analysissupporting

confidence: 74%

Section: A Odorant Identification As Linear Separation Of Curves In N...mentioning

confidence: 99%

Section: A Odorant Identification As Linear Separation Of Curves In N...mentioning

confidence: 99%

See 1 more Smart Citation

Computational Capacity of an Odorant Discriminator: The Linear Separability of Curves

et al. 2002

View full text Add to dashboard Cite

show abstract

“…Both these corrections are of order O(1/ √ n). This behaviour, numerically verified within several learning scenarios (Buhot, Torres Moreno, & Gordon, 1997;Nadler & Fink, 1997;Schroder & Urbanczik, 1998), shows that the predictions of the statistical mechanics approach are better for larger n.…”

Section: Statistical Mechanicsmentioning

confidence: 84%

Untitled

Risau-Gusmán

Gordon

2002

Machine Learning

View full text Add to dashboard Cite

Abstract.We study the typical properties of polynomial Support Vector Machines within a Statistical Mechanics approach that takes into account the number of high order features relative to the input space dimension. We analyze the effect of different features' normalizations on the generalization error, for different kinds of learning tasks. If the normalization is adequately selected, hierarchical learning of features of increasing order takes place as a function of the training set size. Otherwise, the performance worsens, and there is no hierarchical learning at all.

show abstract

“…The challenge of numerically estimating the critical capacity α c has been attacked by several groups, most of them verifying α c ≃ 0.833, with the exception of Ref. [178], criticized by the comment in [179]. Subsequent, more extensive computations in [180,181] appear to confirm the original critical value.…”

Section: Ising Couplingsmentioning

confidence: 99%

Techniques of replica symmetry breaking and the storage problem of the McCulloch–Pitts neuron

Györgyi

2001

Physics Reports

View full text Add to dashboard Cite

In this article we review the framework for spontaneous replica symmetry breaking. Subsequently that is applied to the example of the statistical mechanical description of the storage properties of a McCulloch-Pitts neuron, i. e., simple perceptron. It is shown that in the neuron problem the general formula appears that is at the core of all problems admitting Parisi's replica symmetry breaking ansatz with a one-component order parameter. The details of Parisi's method are reviewed extensively, with regard to the wide range of systems where the method may be applied. Parisi's partial differential equation and related differential equations are discussed, and a Green function technique introduced for the calculation of replica averages, the key to determining the averages of physical quantities. The Green functions of the Fokker-Planck equation due to Sompolinsky turns out to play the role of the statistical mechanical Green function in the graph rules for replica correlators. The subsequently obtained graph rules involve only tree graphs, as appropriate for a mean-field-like model. The lowest order Ward-Takahashi identity is recovered analytically and shown to lead to the Goldstone modes in continuous replica symmetry breaking phases. The need for a replica symmetry breaking theory in the storage problem of the neuron has arisen due to the thermodynamical instability of formerly given solutions. Variational forms for the neuron's free energy are derived in terms of the order parameter function x(q), for different prior distribution of synapses. Analytically in the high temperature limit and numerically in generic cases various phases are identified, among them one similar to the Parisi phase in long range interaction spin glasses. Extensive quantities like the error per pattern change slightly with respect to the known unstable solutions, but there is a significant difference in the distribution of non-extensive quantities like the synaptic overlaps and the pattern storage stability parameter. A simulation result is also reviewed and compared to the prediction of the theory. 07.05. Mh, 75.10Nr, 84.35.+i

show abstract

Finite Size Scaling in Neural Networks

Cited by 6 publications

References 23 publications

Computational Capacity of an Odorant Discriminator: The Linear Separability of Curves

Computational Capacity of an Odorant Discriminator: The Linear Separability of Curves

Untitled

Techniques of replica symmetry breaking and the storage problem of the McCulloch–Pitts neuron

Contact Info

Product

Resources

About