Divergence-Based Vector Quantization

Villmann, Thomas; Haase, Sven

doi:10.1162/neco_a_00110

Cited by 65 publications

(45 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In many applications, such as image analysis, pattern recognition and statistical machine learning we use the information-theoretic divergences rather than Euclidean squared or l p -norm distances [28]. Several information divergences such as Kullback-Leibler, Hellinger and Jensen-Shannon divergences are central to estimate similarity between distributions and have long history in information geometry.…”

Section: D(p || Z) ≤ D(p || Q) + D(q || Z) (Subaddivity/triangle Ineqmentioning

confidence: 99%

“…Recently, alternative generalized divergences such as the Csiszár-Morimoto f -divergence and Bregman divergence become attractive alternatives for advanced machine learning algorithms [26][27][28][29][30][31][32][33][34]. In this paper, we discuss a robust parameterized subclass of the Csiszár-Morimoto and the Bregman divergences: Alpha-and Beta-divergences that may provide more robust solutions with respect to outliers and additive noise and improved accuracy.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Families of Alpha- Beta- and Gamma- Divergences: Flexible and Robust Measures of Similarities

Cichocki

Амари

2010

Entropy

369

309

View full text Add to dashboard Cite

In this paper, we extend and overview wide families of Alpha-, Beta-and Gamma-divergences and discuss their fundamental properties. In literature usually only one single asymmetric (Alpha, Beta or Gamma) divergence is considered. We show in this paper that there exist families of such divergences with the same consistent properties. Moreover, we establish links and correspondences among these divergences by applying suitable nonlinear transformations. For example, we can generate the Beta-divergences directly from Alpha-divergences and vice versa. Furthermore, we show that a new wide class of Gamma-divergences can be generated not only from the family of Beta-divergences but also from a family of Alpha-divergences. The paper bridges these divergences and shows also their links to Tsallis and Rényi entropies. Most of these divergences have a natural information theoretic interpretation.

show abstract

Section: D(p || Z) ≤ D(p || Q) + D(q || Z) (Subaddivity/triangle Ineqmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Families of Alpha- Beta- and Gamma- Divergences: Flexible and Robust Measures of Similarities

Cichocki

Амари

2010

Entropy

369

309

View full text Add to dashboard Cite

show abstract

“…Obviously, the Euclidean distance is not based on a functional norm [2,3,23]. Yet, the transfer to real functional norms and distances like Sobolev norms [24,25], the Lee-norm [23,1], kernel based LVQ-approaches [26] or divergence based similarity measures [27,28], which carry the functional aspect inherently, is straightforward and topic of future investigations.…”

Section: Resultsmentioning

confidence: 99%

“…Thereafter, g S ðtÞ is slowly increased in an adiabatic manner [17], such that all parameters can persistently follow the drift of the system. An additional term for b l -adaptation occurs for non-vanishing g S ðtÞ-values according to this new cost function (27):…”

Section: Structural Sparsitymentioning

confidence: 99%

Functional relevance learning in generalized learning vector quantization

et al. 2012

Self Cite

View full text Add to dashboard Cite

Citation for published version (APA): Kästner, M., Hammer, B., Biehl, M., & Villmann, T. (2012). Functional relevance learning in generalized learning vector quantization. Neurocomputing, 90, 85-95. DOI: 10.1016/j.neucom.2011 Copyright Other than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).Take-down policy If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons the number of authors shown on this cover page is limited to 10 maximum. b s t r a c tRelevance learning in learning vector quantization is a central paradigm for classification task depending feature weighting and selection. We propose a functional approach to relevance learning for high-dimensional functional data. For this purpose we compose the relevance profile by a superposition of only a few parametrized basis functions taking into account the functional character of the data. The number of these parameters is usually significantly smaller than the number of relevance weights in standard relevance learning, which is the number of data dimensions. Thus, instabilities in learning are avoided and an inherent regularization takes place. In addition, we discuss strategies to obtain sparse relevance models for further model optimization.

show abstract

“…It is interesting to consider other potential applications of Hölder divergences and compare their efficiency against the reference Cauchy-Schwarz divergence: For example, HD t-SNE (Stochastic Neighbor Embedding) compared to CS t-SNE [39], HD vector quantization (VQ) compared to CS VQ [40], HD saliency vs. CS saliency detection in images [41], etc.…”

mentioning

confidence: 99%

On Hölder Projective Divergences

Nielsen

Sun

Marchand-Maillet

2017

Entropy

View full text Add to dashboard Cite

Abstract:We describe a framework to build distances by measuring the tightness of inequalities and introduce the notion of proper statistical divergences and improper pseudo-divergences. We then consider the Hölder ordinary and reverse inequalities and present two novel classes of Hölder divergences and pseudo-divergences that both encapsulate the special case of the Cauchy-Schwarz divergence. We report closed-form formulas for those statistical dissimilarities when considering distributions belonging to the same exponential family provided that the natural parameter space is a cone (e.g., multivariate Gaussians) or affine (e.g., categorical distributions). Those new classes of Hölder distances are invariant to rescaling and thus do not require distributions to be normalized. Finally, we show how to compute statistical Hölder centroids with respect to those divergences and carry out center-based clustering toy experiments on a set of Gaussian distributions which demonstrate empirically that symmetrized Hölder divergences outperform the symmetric Cauchy-Schwarz divergence.

show abstract

Divergence-Based Vector Quantization

Cited by 65 publications

References 40 publications

Families of Alpha- Beta- and Gamma- Divergences: Flexible and Robust Measures of Similarities

Families of Alpha- Beta- and Gamma- Divergences: Flexible and Robust Measures of Similarities

Functional relevance learning in generalized learning vector quantization

On Hölder Projective Divergences

Contact Info

Product

Resources

About