Techniques of replica symmetry breaking and the storage problem of the McCulloch–Pitts neuron

Abstract

In this article we review the framework for spontaneous replica symmetry breaking. Subsequently that is applied to the example of the statistical mechanical description of the storage properties of a McCulloch-Pitts neuron, i. e., simple perceptron. It is shown that in the neuron problem the general formula appears that is at the core of all problems admitting Parisi's replica symmetry breaking ansatz with a one-component order parameter. The details of Parisi's method are reviewed extensively, with regard to the wide range of systems where the method may be applied. Parisi's partial differential equation and related differential equations are discussed, and a Green function technique introduced for the calculation of replica averages, the key to determining the averages of physical quantities. The Green functions of the Fokker-Planck equation due to Sompolinsky turns out to play the role of the statistical mechanical Green function in the graph rules for replica correlators. The subsequently obtained graph rules involve only tree graphs, as appropriate for a mean-field-like model. The lowest order Ward-Takahashi identity is recovered analytically and shown to lead to the Goldstone modes in continuous replica symmetry breaking phases. The need for a replica symmetry breaking theory in the storage problem of the neuron has arisen due to the thermodynamical instability of formerly given solutions. Variational forms for the neuron's free energy are derived in terms of the order parameter function x(q), for different prior distribution of synapses. Analytically in the high temperature limit and numerically in generic cases various phases are identified, among them one similar to the Parisi phase in long range interaction spin glasses. Extensive quantities like the error per pattern change slightly with respect to the known unstable solutions, but there is a significant difference in the distribution of non-extensive quantities like the synaptic overlaps and the pattern storage stability parameter. A simulation result is also reviewed and compared to the prediction of the theory. 07.05. Mh, 75.10Nr, 84.35.+i

Evolutionary games on graphs

“…About minority games, as mentioned previously, the reader can find detailed discussions in the books by Challet et al (2004) and by Coolen (2005). Methods developed originally within spin glass theory [for reviews see (Mézard et al, 1987;Györgyi, 2001)] are used successfully by Berg and Engel (1998); Berg and Weigt (1999), who studied the properties of two-player games with random payoffs in the limit Q → ∞.…”

Out-of-equilibrium dynamical mean-field equations for the perceptron model

“…This model shares most of the basic phenomenology of p-spin glasses (in particular the RFOT phenomenology), but it also displays additional interesting features that characterise particle systems, such as a Gardner and a jamming transition [33][34][35]. Perceptron models, introduced by McCulloch and Pitts as simple models of a neuron [36], and later proposed by Rosenblatt [37] as the simplest unit of a learning machine [38,39], appear mutatis mutandis as the building blocks of many theories in a broad range of different fields of science. The perceptron problem can be seen as the simplest classification task: given a set of inputs or patterns and a set of associated outputs, one wants to find the synaptic weights such that the input patterns are correctly classified as the prescribed outputs.…”

A nonextensive method for spectroscopic data analysis with artificial neural networks

“…However, the drawback of this class of methods is that they are very computationally expensive. Statistical mechanical methods have also been successfully applied to the study of neural network models of associative memory [12]. Another class of efficient training algorithms is the conjugate gradient learning based schemes.…”