Projection Theorems for the Rényi Divergence on $\alpha $ -Convex Sets

Abstract

This paper studies forward and reverse projections for the Rényi divergence of order α ∈ (0, ∞) on α-convex sets. The forward projection on such a set is motivated by some works of Tsallis et al. in statistical physics, and the reverse projection is motivated by robust statistics. In a recent work, van Erven and Harremoës proved a Pythagorean inequality for Rényi divergences on α-convex sets under the assumption that the forward projection exists. Continuing this study, a sufficient condition for the existence of a forward projection is proved for probability measures on a general alphabet. For α ∈ (1, ∞), the proof relies on a new Apollonius theorem for the Hellinger divergence, and for α ∈ (0, 1), the proof relies on the Banach-Alaoglu theorem from functional analysis. Further projection results are then obtained in the finite alphabet setting. These include a projection theorem on a specific α-convex set, which is termed an α-linear family, generalizing a result by Csiszár to α = 1. The solution to this problem yields a parametric family of probability measures which turns out to be an extension of the exponential family, and it is termed an α-exponential family. An orthogonality relationship between the α-exponential and α-linear families is established, and it is used to turn the reverse projection on an α-exponential family into a forward projection on an α-linear family. This paper also proves a convergence result of an iterative procedure used to calculate the forward projection on an intersection of a finite number of α-linear families.

Rate Distortion Theory for Descriptive Statistics

“…Second, even if D(P C) = ∞, the Rényi divergence D α (P Q) (see e.g. [20]) may be a well-defined non-negative real number for α ∈ (0, 1) and Q ∈ C. These Rényi divergences are jointly convex in P and Q [20] and for each 0 < α < 1 one may define a reversed Rényi projection Qα of P on C [21]. If it exists, it can be shown that this distribution will satisfy…”