Learning vector quantization: The dynamics of winner-takes-all algorithms

Biehl, Michael; Ghosh, Anarta; Hammer, Barbara

doi:10.1016/j.neucom.2005.12.007

Cited by 30 publications

(17 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…offset of the clusters σ , variance within the clusters υ σ , learning rate η, and for NG, the rank function parameter λ. As shown in [5], this method of analysis is in good agreement with large scale Monte Carlo simulations of the same learning systems for dimensionality as low as N = 200.…”

Section: Analysis Of Learning Dynamicssupporting

confidence: 70%

“…This successful approach has also been reviewed in [8,16], among others. The extension of the theoretical analysis of simple (WTA-based) vector quantization with two prototypes and two clusters introduced in an earlier work [5] is not straightforward. Additional prototypes and clusters introduce more complex interactions in the system that can result in radically different behaviors.…”

Section: Introductionmentioning

confidence: 98%

See 1 more Smart Citation

Learning dynamics and robustness of vector quantization and neural gas

et al. 2008

Self Cite

View full text Add to dashboard Cite

Various alternatives have been developed to improve the Winner-Takes-All (WTA) mechanism in vector quantization, including the Neural Gas (NG). However, the behavior of these algorithms including their learning dynamics, robustness with respect to initialization, asymptotic results, etc. has only partially been studied in a rigorous mathematical analysis. The theory of on-line learning allows for an exact mathematical description of the training dynamics in model situations. We demonstrate using a system of three competing prototypes trained from a mixture of Gaussian clusters that the Neural Gas can improve convergence speed and achieves robustness to initial conditions. However, depending on the structure of the data, the Neural Gas does not always obtain the best asymptotic quantization error.

show abstract

Section: Analysis Of Learning Dynamicssupporting

confidence: 70%

Section: Introductionmentioning

confidence: 98%

Learning dynamics and robustness of vector quantization and neural gas

et al. 2008

Self Cite

View full text Add to dashboard Cite

show abstract

“…We only mention that unsupervised prototype based learning has been treated in complete analogy to the above [43,44]. Technically, it reduces to the consideration of modulation functions which do not depend on the cluster or class label.…”

Section: Dynamics Of Prototype Based Learningmentioning

confidence: 99%

“…The basic competitive WTA Vector Quantization training would be represented, for instance, by the modulation function f s = Θ(d −s −d +s ) which always moves the winning prototype closer to the data. The training prescription can be interpreted as a stochastic gradient descent of a cost function, the so-called quantization error [44]. The exchange and permutation symmetry of prototypes in unsupervised training results in interesting effects which resemble the plateaus discussed in multilayered neural networks, cf.…”

Section: Dynamics Of Prototype Based Learningmentioning

confidence: 99%

Statistical Mechanics of On-line Learning

Biehl

Caticha

Riegler

2009

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Abstract. We introduce and discuss the application of statistical physics concepts in the context of on-line machine learning processes. The consideration of typical properties of very large systems allows to perfom averages over the randomness contained in the sequence of training data. It yields an exact mathematical description of the training dynamics in model scenarios. We present the basic concepts and results of the approach in terms of several examples, including the learning of linear separable rules, the training of multilayer neural networks, and Learning Vector Quantization.

show abstract

“…This problem relates to unexpected behavior and instabilities of training. It has been shown that already slight variations of the basic LVQ learning scheme yield quite different results [2,3]. Variants of LVQ which can be derived from an explicit cost function are particularly interesting.…”

Section: Introductionmentioning

confidence: 99%

Adaptive Relevance Matrices in Learning Vector Quantization

2009

Self Cite

View full text Add to dashboard Cite

We propose a new matrix learning scheme to extend relevance learning vector quantization (RLVQ), an efficient prototype-based classification algorithm, towards a general adaptive metric. By introducing a full matrix of relevance factors in the distance measure, correlations between different features and their importance for the classification scheme can be taken into account and automated, general metric adaptation takes place during training. In comparison to the weighted Euclidean metric used in RLVQ and its variations, a full matrix is more powerful to represent the internal structure of the data appropriately. Large margin generalization bounds can be transfered to this case leading to bounds which are independent of the input dimensionality. This also holds for local metrics attached to each prototype which corresponds to piecewise quadratic decision boundaries. The algorithm is tested in comparison to alternative LVQ schemes using an artificial data set, a benchmark multi-class problem from the UCI repository, and a problem from bioinformatics, the recognition of splice sites for C. elegans.

show abstract

Learning vector quantization: The dynamics of winner-takes-all algorithms

Cited by 30 publications

References 7 publications

Learning dynamics and robustness of vector quantization and neural gas

Learning dynamics and robustness of vector quantization and neural gas

Statistical Mechanics of On-line Learning

Adaptive Relevance Matrices in Learning Vector Quantization

Contact Info

Product

Resources

About