$f$ -Divergence Inequalities

Sason, Igal; Verdú, S.

doi:10.1109/tit.2016.2603151

Cited by 248 publications

(252 citation statements)

References 96 publications

Supporting

Mentioning

250

Contrasting

Order By: Relevance

“…As a divergence, generalized reliability has connections with other forms of divergence that help to characterize its features and translate between different inferential paradigms. Consider, for example, the variation distance, another type of f-divergence (with f(u) = |u-1| in Equation 3; Gilardoni, 2010;Sason & Verdu, 2016):…”

Section: Relationships With Other Forms Of Divergencementioning

confidence: 99%

“…A number of inequalities relating variational distance to relative entropy have been identified (Gilardoni;Sason & Verdu, 2016). The most well-known of these are Pinsker-style inequalities, having the form…”

Section: Relationships With Other Forms Of Divergencementioning

confidence: 99%

“…As such, it is a special case of f-divergence (Amari, 2009;Gilardoni, 2010;Liese & Vajda, 2006;Sason & Verdu, 2016), which can be defined as…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Reliability as Lindley Information

Ke¹

2017

Preprint

View full text Add to dashboard Cite

This paper introduces a generalized definition of reliability based on Lindley information, which is the mutual information between an observed measure and latent attribute. This definition reduces to the traditional definition of reliability in the case of normal variables, but can be applied to any joint distribution of observed and latent variables. Importantly, unlike traditional definitions of reliability, this formulation of reliability applies to vector-or matrix-valued estimates and summaries of responses, and therefore generalizes reliability to sets of scores and estimates in addition to individual scores and estimates. This formulation also leads to new bounds for reliability, as well as newly reported relationships between reliability and the traditional Fisher information function familiar in item response theory literature. The generalized reliability can be estimated using formulae, or methods used in Bayesian inference such as Markov Chain Monte Carlo (MCMC) depending on the case. Examples based on well-studied datasets are provided, as well as an application to randomly-varying intraindividual covariance structures.Running head: GENERALIZED RELIABILITY 3 The aim of this paper is to introduce a generalized definition of reliability given aswhere ι is the Lindley information (Lindley, 1956), which in this context is the mutual information between an observed variable X and a latent attribute θ:Either X or θ could be multidimensional.Lindley information can be interpreted various ways depending on inferential paradigm and perspective: as an index of the expected change in probability of X and θ under knowledge of their joint distribution, relative to knowledge only of their independent marginal distributions; as the expected change in probability of θ given the data, relative to the prior distribution of θ; as the expected per-observation log-likelihood ratio comparing a model of measure informativeness to a model of measure uninformativeness (Kinney & Atwal, 2014); or as the difference in codelength required to describe the data and θ under a model of informativeness compared to a model of uninformativeness.More generally, Lindley information can be interpreted as the divergence of the joint observed-latent distribution p(X, θ) from the independence distribution p(X)p(θ). As such, it is a special case of f-divergence (Amari, 2009;Gilardoni, 2010;Liese & Vajda, 2006;Sason & Verdu, 2016), which can be defined aswhere f(u) is a convex function f:(0, ∞) → , and p and q are probability densities absolutely continuous with ℝ regard to a reference measure ν. Lindley information is obtained by setting f(u) = u ln u, with p = p(X, θ) and q = p(X)p(θ), and is equivalent to the relative entropy or Kullback-Liebler divergence between p(X, θ) and p(X)p(θ).Like traditional reliability, the generalized definition of reliability implied by Equations 1 and 2 produces a statistic that generally ranges between 0 and 1, and reflects the measurement precision of an indicator, such that values closer to 1 indic...

show abstract

Section: Relationships With Other Forms Of Divergencementioning

confidence: 99%

Section: Relationships With Other Forms Of Divergencementioning

confidence: 99%

See 1 more Smart Citation

Reliability as Lindley Information

Ke¹

2017

Preprint

View full text Add to dashboard Cite

show abstract

“…2 A simple proof of the monotonicity of the Hellinger divergence in α appears in [35,Theorem 33]. 8 Notice that the multiplication of p α by the scalar s ∈ [0, 1] in the right side of (27) does not affect the supremum in (26).…”

Section: Proofmentioning

confidence: 99%

Projection Theorems for the Rényi Divergence on $\alpha $ -Convex Sets

Kumar

Sason

2016

IEEE Trans. Inform. Theory

Self Cite

View full text Add to dashboard Cite

This paper studies forward and reverse projections for the Rényi divergence of order α ∈ (0, ∞) on α-convex sets. The forward projection on such a set is motivated by some works of Tsallis et al. in statistical physics, and the reverse projection is motivated by robust statistics. In a recent work, van Erven and Harremoës proved a Pythagorean inequality for Rényi divergences on α-convex sets under the assumption that the forward projection exists. Continuing this study, a sufficient condition for the existence of a forward projection is proved for probability measures on a general alphabet. For α ∈ (1, ∞), the proof relies on a new Apollonius theorem for the Hellinger divergence, and for α ∈ (0, 1), the proof relies on the Banach-Alaoglu theorem from functional analysis. Further projection results are then obtained in the finite alphabet setting. These include a projection theorem on a specific α-convex set, which is termed an α-linear family, generalizing a result by Csiszár to α = 1. The solution to this problem yields a parametric family of probability measures which turns out to be an extension of the exponential family, and it is termed an α-exponential family. An orthogonality relationship between the α-exponential and α-linear families is established, and it is used to turn the reverse projection on an α-exponential family into a forward projection on an α-linear family. This paper also proves a convergence result of an iterative procedure used to calculate the forward projection on an intersection of a finite number of α-linear families.

show abstract

“…For two probability measures P and Q, divergences satisfy the following properties. In particular, the f -divergence [15], [7], the Bregman divergence [3] and the Rényi divergence [16], [14] are often used in various fields such as machine learning, image processing, statistical physics, finance and so on.…”

Section: Introductionmentioning

confidence: 99%