Generalizing with perceptrons in the case of structured phase- and pattern-spaces

Abstract

We investigate the influence of different kinds of structure on the learning behaviour of a perceptron performing a classification task defined by a teacher rule. The underlying pattern distribution is permitted to have spatial correlations. The prior distribution for the teacher coupling vectors itself is assumed to be nonuniform. Thus classification tasks of quite different difficulty are included. As learning algorithms we discuss Hebbian learning, Gibbs learning, and Bayesian learning with different priors, using methods from statistics and the replica formalism. We find that the Hebb rule is quite sensitive to the structure of the actual learning problem, failing asymptotically in most cases. Contrarily, the behaviour of the more sophisticated methods of Gibbs and Bayes learning is influenced by the spatial correlations only in an intermediate regime of α, where α specifies the size of the training set. Concerning the Bayesian case we show, how enhanced prior knowledge improves the performance.

Structured Prediction in NLP -- A survey

“…Structured Perceptron [6] and structured SVM [7] based approaches only require efficiently compute the arg max, unlike CRFs which require computing many other marginals and partition function. These approaches are non-probabilistic, and they utilize a simple form for the score:…”

Classification of Correlated Patterns with a Configurable Analog VLSI Neural Network of Spiking Neurons and Self-Regulating Plastic Synapses

“…In particular, correlations in the patterns (the fact that the components of the pattern vectors are not chosen independently from each other) pose problems whenever, for example, the network is to classify similar patterns into different classes. The implications of correlations in the patterns structure have been studied in different contexts, including correlation effects arising from varying the coding level, which is the fraction of active neurons for each pattern (Amit & Fusi, 1994;Tsodyks & Feigelman, 1988), as well as the implications of spatial correlations (between components of the patterns) or semantic correlations (between different patterns) (Tarkowski & Lewenstein, 1993;Lopez, Schroder, & Opper, 1995;Dirscherl, Schottky, & Krey, 1998).…”