Computational complexity, learning rules and storage capacities: A Monte Carlo study for the binary perceptron

Patel, Hetal

doi:10.1007/bf01315244

Cited by 14 publications

(25 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…equal to one if predicate is true and zero otherwise). By setting κ = 0.5 this objective function was found to more effective than the naive choice-this is in accordance with the findings in reference [29]. In figure 10, we show the performance of a hillclimber with and without averaging.…”

Section: B Binary Perceptronsupporting

confidence: 86%

“…This problem has been very heavily analysed [24], [25], [26], [27]. It is known to be NP-Hard [28], but more significantly, in practice, it has proved to be extremely difficult to solve due to the enormous number of local minima it has [29], [11], [30]. Rather than perform hill-climbing using the number of misclassified patterns (which gives very poor performance) we instead minimise…”

Section: B Binary Perceptronmentioning

confidence: 99%

See 1 more Smart Citation

Learning the Large-Scale Structure of the MAX-SAT Landscape Using Populations

Qasem

Prügel-Bennett

2010

IEEE Trans. Evol. Computat.

View full text Add to dashboard Cite

Abstract-A new algorithm for solving MAX-SAT problems is introduced which clusters good solutions, and restarts the search from the closest feasible solution to the centroid of each cluster. This is shown to be highly efficient for finding good solutions of large MAX-SAT problems. We argue that this success is due to the population learning the large-scale structure of the fitness landscape. Systematic studies of the landscape are presented to support this hypothesis. In addition, a number of other strategies are tested to rule out other possible explanations of the success.Preliminary results are shown indicating that extensions of the proposed algorithm can give similar improvements on other hard optimisation problems.

show abstract

Section: B Binary Perceptronsupporting

confidence: 86%

Section: B Binary Perceptronmentioning

confidence: 99%

Learning the Large-Scale Structure of the MAX-SAT Landscape Using Populations

Qasem

Prügel-Bennett

2010

IEEE Trans. Evol. Computat.

View full text Add to dashboard Cite

show abstract

“…Simulation studies of the single-layer binary perceptron have been performed before for the problems of storing binary [16][17][18][19]22,23], and Gaussian patterns [22,24], using various approaches and not always leading to conclusive results. Our result for α c differs significantly from the analytical result of Ref.…”

mentioning

confidence: 99%

Finite Size Scaling in Neural Networks

Nadler

Fink

1997

Phys. Rev. Lett.

View full text Add to dashboard Cite

We demonstrate that the fraction of pattern sets that can be stored in single-and hidden-layer perceptrons exhibits finite size scaling. This feature allows to estimate the critical storage capacity αc from simulations of relatively small systems. We illustrate this approach by determining αc, together with the finite size scaling exponent ν, for storing Gaussian patterns in committee and parity machines with binary couplings and up to K = 5 hidden units. 87.10.+e, 64.60.Cn, 05.50.+q, 02.70.Lq Finite size scaling (FSS) has proven to be a powerful method for analyzing phase transitions, which occur rigorously only in the thermodynamic limit, using simulations of systems of finite size [1]. In particular, it has become the prime method for determining numerical values of critical coupling parameters and exponents [2].Phase transitions are known to occur not only in condensed matter [3] and percolation systems [2], but also in random graphs [4], neural networks [5], and in algorithmic problems like search [6] and the satisfiability of random boolean expressions [7]. Heuristic derivations of FSS rely on the divergence of a correlation length at a critical point in the infinite system [2,8]. However, Kirkpatrick and Selman [9] have demonstrated recently that FSS can be used efficiently also in problems without any intrinsic length scales, like the connectivity of random graphs and the satisfiability of random boolean expressions. Abstract neural networks [5] are another class of systems without intrinsic length scale, and we will show in this contribution that FSS occurs at the transition from storable to unstorable pattern set sizes, and that it provides a powerful computational method for determining critical storage capacities.We will concentrate on particular feed-forward networks of the perceptron class, namely multi-layer perceptrons with N input neurons, K hidden units, and a regular tree-like connectivity (N mod K = 0), see Fig.

show abstract

“…Above the freezing temperature there is a high probability of the Monte Carlo algorithm accepting a move, while below the freezing temperature the searcher gets trapped in a local optimum with an exponentially small probability of escaping. It is found that a good annealing schedule for many problems involves setting the temperature to just above the freezing temperature [31], [32]. Many heuristics have been developed within the simulated annealing community to choose good annealing schedules based on the performance of the searcher.…”

Section: Parameter Tuningmentioning

confidence: 99%

Benefits of a Population: Five Mechanisms That Advantage Population-Based Algorithms

Prügel-Bennett

2010

IEEE Trans. Evol. Computat.

View full text Add to dashboard Cite

Abstract-This paper identifies five distinct mechanisms by which a population-based algorithm might have an advantage over a solo-search algorithm in classical optimisation. These mechanisms are illustrated through a number of toy problems. Simulations are presented comparing different search algorithms on these problems. The plausibility of these mechanisms occurring in classical optimisation problems is discussed.The first mechanism we consider relies on putting together building blocks from different solutions. This is extended to include problems containing critical variables. The second mechanism is the result of focusing of the search caused by crossover. Also discussed in this context is strong focusing produced by averaging many solutions. The next mechanism to be examined is the ability of a population to act as a low-pass filter of the landscape, ignoring local distractions. The fourth mechanism is a population's ability to search different parts of the fitness landscape, thus hedging against bad luck in the initial position or the decisions it makes. The final mechanism is the opportunity of learning useful parameter values to balance exploration against exploitation.

show abstract

Computational complexity, learning rules and storage capacities: A Monte Carlo study for the binary perceptron

Cited by 14 publications

References 32 publications

Learning the Large-Scale Structure of the MAX-SAT Landscape Using Populations

Learning the Large-Scale Structure of the MAX-SAT Landscape Using Populations

Finite Size Scaling in Neural Networks

Benefits of a Population: Five Mechanisms That Advantage Population-Based Algorithms

Contact Info

Product

Resources

About