On Relations Between the Relative Entropy and χ2-Divergence, Generalizations and Applications

Abstract

The relative entropy and the chi-squared divergence are fundamental divergence measures in information theory and statistics. This paper is focused on a study of integral relations between the two divergences, the implications of these relations, their information-theoretic applications, and some generalizations pertaining to the rich class of f-divergences. Applications that are studied in this paper refer to lossless compression, the method of types and large deviations, strong data–processing inequalities, bounds on contraction coefficients and maximal correlation, and the convergence rate to stationarity of a type of discrete-time Markov chains.

Relative Entropy as a Measure of Difference between Hermitian and Non-Hermitian Systems

“…Here, we propose that comparing eigenmodes is a more generic and appropriate method for describing a difference between Hermitian and non-Hermitian systems. For this reason, we introduce the notion of relative entropy (also known as the Kullback–Leibler divergence) [ 26 , 27 ]. The relative entropy is a measure of the difference in two probability distributions on the probability space.…”

“…Accordingly, we need to overcome this problem considering a different disparity, not of eigenvalues but of the eigenmodes in open systems. For this, we exploit the notion of relative entropy [ 26 , 27 ], which is typically used to measure a difference given in the form of two probability distribution functions, to quantify the difference between the Hermitian and non-Hermitian eigenmodes. We could expect that adapting the relative entropy can be advantageous for future works on optical microcavity as in the cases of information theory.…”

Strongly Convex Divergences

“…We need the following lemma originally proved by Audenart in the quantum setting [ 28 ]. It is based on a differential relationship between the skew divergence [ 12 ] and the [ 15 ] (see [ 29 , 30 ]).…”

A Two-Moment Inequality with Applications to Rényi Entropy and Mutual Information

“…Let P and Q be distributions defined on a common probability space that have densities p and q with respect to a dominating measure . The relative entropy (or Kullback–Leibler divergence) is defined according to and the chi-squared divergence is defined as Both of these divergences can be seen as special cases of the general class of f -divergence measures and there exists a rich literature on comparisons between different divergences [ 8 , 26 , 27 , 28 , 29 , 30 , 31 , 32 ]. The chi-squared divergence can also be viewed as the squared distance between and .…”

“…The chi-square can also be interpreted as the first non-zero term in the power series expansion of the relative entropy ([ 26 ], [Lemma 4]). More generally, the chi-squared divergence provides an upper bound on the relative entropy via The proof of this inequality follows straightforwardly from Jensen’s inequality and the concavity of the logarithm; see [ 27 , 31 , 32 ] for further refinements.…”