On Bregman Distances and Divergences of Probability Measures

Abstract

The paper introduces scaled Bregman distances of probability distributions which admit non-uniform contributions of observed events. They are introduced in a general form covering not only the distances of discrete and continuous stochastic observations, but also the distances of random processes and signals. It is shown that the scaled Bregman distances extend not only the classical ones studied in the previous literature, but also the information divergence and the related wider class of convex divergences of probability measures. An information processing theorem is established too, but only in the sense of invariance w.r.t. statistically sufficient transformations and not in the sense of universal monotonicity. Pathological situations where coding can increase the classical Bregman distance are illustrated by a concrete example. In addition to the classical areas of application of the Bregman distances and convex divergences such as recognition, classification, learning and evaluation of proximity of various features and signals, the paper mentions a new application in 3D-exploratory data analysis. Explicit expressions for the scaled Bregman distances are obtained in general exponential families, with concrete applications in the binomial, Poisson and Rayleigh families, and in the families of exponential processes such as the Poisson and diffusion processes including the classical examples of the Wiener process and geometric Brownian motion.

Of Quantiles and Expectiles: Consistent Scoring Functions, Choquet Representations and Forecast Rankings

“…Another setting arises when higher dimensional aspects are compressed into real‐valued functionals, such as in the case of ϕ ‐divergences of probability measures P and Q . In a nutshell, a ϕ ‐divergence is of the formwhere ϕ is a convex function on [0, ∞) such that ϕ (1)=0; see, for example Stummer and Vajda () for a recent treatment and a delineation from Bregman distances. Any function ϕ of this form admits a mixture representation in terms of elementary functions or atoms for θ >0.…”

A new toolkit for robust distributional change detection

“…To embed the abovementioned two major divergence classes as a special case of Definition , let us first mention Stummer and Stummer and Vajda have shown that in case of ϕ (1) = 0 for the choice m ( y ): = q ( y ) ( ) the scaled Bregman distance becomes which is nothing but the well‐known ϕ ‐divergence between P and Q (and on the right‐hand side, one has to additionally subtract ϕ (1) in case that it is not zero). The latter has been first studied by Csiszár and Ali and Silvey .…”

Scaled Bregman divergences in a Tsallis scenario

“…Scaled Bregman divergences, formally introduced by Stummer [24] and Stummer and Vajda [25], unify separable Bregman divergences [10] (defined below in Section 2.3) and f-divergences [8,9]. This paper uses scaled Bregman divergences as its basis, and accomplishes the following objectives:…”

“…Definition 2 (Notations) [25]: M denotes the space of all finite measures on a measurable space (X , A) and P ⊂ M the subspace of all probability measures. Unless otherwise explicitly stated P,R,M are mutually measure-theoretically equivalent measures on (X , A) dominated by a σ-finite measure λ on (X , A).…”

“…[24] and [25], when extended to deformed statistics. The above result may also be employed in the case of the dual generalized K-Ld between a conditional probability and a marginal probability.…”