The Kullback–Leibler Divergence Rate Between Markov Sources

Rached, Z.; Alajaji, Fady; Campbell, L. L.

doi:10.1109/tit.2004.826687

Cited by 110 publications

(73 citation statements)

References 15 publications

(12 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More specifically, it is sufficient to show the following inequality (14) holds for each 1 ≤ j ≤ q. Then an upper bound of ν(G x , G y ) as given in (12) will be smaller than (13), which is the right hand side of (10). For a finite p, the right hand side of (14) is given by…”

Section: '"mentioning

confidence: 99%

“…Although the K-L divergence does not satisfy the symmetry property and is not a metric, it provides useful interpretations for problems related to probability distributions. The usage of the K-L divergence rate as a measure of distance between two Markov chains has been widely accepted [13], [18]. Therefore, although taking d(·, ·) as the K-L divergence in (5) does not provide a metric, some notion of similarity can still be inferred.…”

Section: A Extension: When D(· ·) Is Not a P-normmentioning

confidence: 99%

“…Yunwen Xu, S. Salapaka and C. L. Beck are from University of Illinois at Urbana-Champaign, Urbana, Illinois. Email: [xu27, salapaka, beck3]@illinois.edu chains [13], and an information-based metric between two probability distributions [14]. Most of these approaches have been developed for undirected unweighted graphs and are not applicable to weighted and/or directed graphs.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A distance metric between directed weighted graphs

Salapaka

Beck

2013

52nd IEEE Conference on Decision and Control

View full text Add to dashboard Cite

Abstract-Directed weighted graphs are increasingly used to model complex systems and interactions, such as networks of interconnected physical or biological subsystems. The analysis of these graphs often requires some form of dissimilarity, or distance measure to compare graphs. In this paper, we extend connectivity-based dissimilarity measures previously used to compare unweighted undirected graphs of the same dimensions to: (1) directed weighted graphs of the same dimensions and (2) directed weighted graphs of different dimensions. To our knowledge, this is the first approach proposed for comparing two graphs containing different numbers of nodes. We derive the conditions under which this dissimilarity measure is a pseudo-metric. This derivation provides new insights on our algorithms (previously proposed) for the graph aggregation optimization problem.

show abstract

Section: '"mentioning

confidence: 99%

Section: A Extension: When D(· ·) Is Not a P-normmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A distance metric between directed weighted graphs

Salapaka

Beck

2013

52nd IEEE Conference on Decision and Control

View full text Add to dashboard Cite

show abstract

“…For instance, where corresponds to the statistical mean under the distribution of . Considering as the initial states distribution of the process, and as the transition matrix (16) where (17) is the Shannon entropy rate of the first-order Markov process [47]. Thus, we can rewrite the log-likelihood ratio test in terms of the Shannon entropy rate of the processes as (18) In general, for and sufficiently large, the first term can be dismissed [46].…”

Section: B Kullback-leibler Merging Criterionmentioning

confidence: 99%

Region Merging Techniques Using Information Theory Statistical Measures

Calderero

Marqués

2010

IEEE Trans. on Image Process.

116

View full text Add to dashboard Cite

Abstract-The purpose of the current work is to propose, under a statistical framework, a family of unsupervised region merging techniques providing a set of the most relevant region-based explanations of an image at different levels of analysis. These techniques are characterized by general and nonparametric region models, with neither color nor texture homogeneity assumptions, and a set of innovative merging criteria, based on information theory statistical measures. The scale consistency of the partitions is assured through i) a size regularization term into the merging criteria and a classical merging order, or ii) using a novel scale-based merging order to avoid the region size homogeneity imposed by the use of a size regularization term. Moreover, a partition significance index is defined to automatically determine the subset of most representative partitions from the created hierarchy. Most significant automatically extracted partitions show the ability to represent the semantic content of the image from a human point of view. Finally, a complete and exhaustive evaluation of the proposed techniques is performed, using not only different databases for the two main addressed problems (object-oriented segmentation of generic images and texture image segmentation), but also specific evaluation features in each case: under-and oversegmentation error, and a large set of region-based, pixel-based and error consistency indicators, respectively. Results are promising, outperforming in most indicators both object-oriented and texture state-of-the-art segmentation techniques.

show abstract

“…An appropriate metric to quantify the distance between two distributions x(i) and z( j) is the K-L divergence [23] given…”

Section: Model Reduction For Markov Chainsmentioning

confidence: 99%

On reduction of graphs and Markov chain models

Salapaka

Beck

2011

IEEE Conference on Decision and Control and European Control Conference

View full text Add to dashboard Cite

Abstract-This paper introduces a new method for reducing large directed graphs to simpler graphs with fewer nodes. The reduction is carried out through node and edge aggregation, where the simpler graph is representative of the original large graph. Representativeness is measured using a metric defined herein, which is motivated by thermodynamic free energy and vector quantization problems in the data compression literature.The resulting aggregation scheme is largely based on the maximum entropy principle. The proposed algorithm is general in the sense that it can accommodate a large class of functions for characterizing distance between the nodes. As a special case, we show that this method applies to the Markov chain model-reduction problem, providing a soft-clustering approach that enables better aggregation of state-transition matrices than existing methods. Simulation results are provided to illustrate the theoretical results.

show abstract

The Kullback–Leibler Divergence Rate Between Markov Sources

Cited by 110 publications

References 15 publications

A distance metric between directed weighted graphs

A distance metric between directed weighted graphs

Region Merging Techniques Using Information Theory Statistical Measures

On reduction of graphs and Markov chain models

Contact Info

Product

Resources

About