Some inequalities for information divergence and related measures of discrimination

Topsøe, Flemming

doi:10.1109/18.850703

Cited by 210 publications

(167 citation statements)

References 25 publications

Supporting

Mentioning

164

Contrasting

Order By: Relevance

“…One way to proceed in this direction is to empirically show a constant factor relationship between these two measurements. In an earlier work by Topsøe [37] it was shown that Jensen-Shannon divergence (referred to as capacitory discrimination) behaves similarly with the triangle divergence (triangular discrimination):…”

Section: Transforming Divergencesmentioning

confidence: 96%

Efficient Nearest-Neighbor Search in the Probability Simplex

Krstovski

Smith

Wallach

et al. 2013

Proceedings of the 2013 Conference on the Theory of Information Retrieval

View full text Add to dashboard Cite

Document similarity tasks arise in many areas of information retrieval and natural language processing. A fundamental question when comparing documents is which representation to use. Topic models, which have served as versatile tools for exploratory data analysis and visualization, represent documents as probability distributions over latent topics. Systems comparing topic distributions thus use measures of probability divergence such as KullbackLeibler, Jensen-Shannon, or Hellinger. This paper presents novel analysis and applications of the reduction of Hellinger divergence to Euclidean distance computations. This reduction allows us to exploit fast approximate nearest-neighbor (NN) techniques, such as locality-sensitive hashing (LSH) and approximate search in k-d trees, for search in the probability simplex. We demonstrate the effectiveness and efficiency of this approach on two tasks using latent Dirichlet allocation (LDA) document representations: discovering relationships between National Institutes of Health (NIH) grants and prior-art retrieval for patents. Evaluation on these tasks and on synthetic data shows that both Euclidean LSH and approximate kd tree search perform well when a single nearest neighbor must be found. When a larger set of similar documents is to be retrieved, the k-d tree approach is more effective and efficient. Categories and Subject Descriptors General TermsExperimentation, Performance Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Permissions@acm.org.

show abstract

Section: Transforming Divergencesmentioning

confidence: 96%

Efficient Nearest-Neighbor Search in the Probability Simplex

Krstovski

Smith

Wallach

et al. 2013

Proceedings of the 2013 Conference on the Theory of Information Retrieval

View full text Add to dashboard Cite

show abstract

“…It has been shown that √ JS satisfies the triangle inequality and that it is an Hilbertian metric [24], [25].…”

Section: A the Jensen-shannon Divergencementioning

confidence: 99%

“…Of course, this includes the denormalized Shannon entropy (3) as a particular case (for q = 1). Although, for the Shannon entropy case, part of the proof is in [27], [25], [23], we present a general proof here.…”

Section: B Jensen-shannon and Tsallis Kernelsmentioning

confidence: 99%

Tsallis kernels on measures

Martins

Aguiar

Figueiredo

2008

2008 IEEE Information Theory Workshop

View full text Add to dashboard Cite

Abstract-Recent approaches to classification of text, images, and other types of structured data, launched the quest for positive definite (p.d.) kernels on probability measures. In particular, kernels based on the Jensen-Shannon (JS) divergence and other information-theoretic quantities have been proposed. We introduce new JS-type divergences, by extending its two building blocks: convexity and Shannon's entropy. These divergences are then used to define new information-theoretic kernels on measures. In particular, we introduce a new concept of qconvexity, for which a Jensen q-inequality is proved. Based on this inequality, we introduce the Jensen-Tsallis q-difference, a nonextensive generalization of the Jensen-Shannon divergence. Furthermore, we provide denormalization formulae for entropies and divergences, which we use to define a family of nonextensive information-theoretic kernels on measures. This family, grounded in nonextensive entropies, extends Jensen-Shannon divergence kernels, and allows assigning weights to its arguments.

show abstract

“…It can be shown that that √ JS satisfies the triangle inequality and is a Hilbertian metric 2 (Endres & Schindelin, 2003;Topsøe, 2000), which has motivated its use in kernel-based machine learning.…”

Section: Jensen-shannon (Js) Divergence Consider a Classification Promentioning

confidence: 99%

“…Of course, this includes the denormalized Shannon entropy as a particular case, corresponding to q = 1. Partial proofs are given by Berg et al (1984), Topsøe (2000), and Cuturi et al (2005); we present here a complete proof.…”

Section: Jensen-shannon and Tsallis Kernelsmentioning

confidence: 99%

Nonextensive Entropic Kernels

Martins¹,

Figueiredo²,

Aguiar³

et al. 2008

View full text Add to dashboard Cite

Positive definite kernels on probability measures have been recently applied in structured data classification problems. Some of these kernels are related to classic information theoretic quantities, such as mutual information and the Jensen-Shannon divergence. Meanwhile, driven by recent advances in Tsallis statistics, nonextensive generalizations of Shannon's information theory have been proposed. This paper bridges these two trends. We introduce the Jensen-Tsallis q-difference, a generalization of the Jensen-Shannon divergence. We then define a new family of nonextensive mutual information kernels, which allow weights to be assigned to their arguments, and which includes the Boolean, Jensen-Shannon, and linear kernels as particular cases. We illustrate the performance of these kernels on text categorization tasks.

show abstract

Some inequalities for information divergence and related measures of discrimination

Cited by 210 publications

References 25 publications

Efficient Nearest-Neighbor Search in the Probability Simplex

Efficient Nearest-Neighbor Search in the Probability Simplex

Tsallis kernels on measures

Nonextensive Entropic Kernels

Contact Info

Product

Resources

About