Re-examination of Bregman functions and new properties of their divergences

Abstract

The Bregman divergence (Bregman distance, Bregman measure of distance) is a certain useful substitute for a distance, obtained from a well-chosen function (the "Bregman function"). Bregman functions and divergences have been extensively investigated during the last decades and have found applications in optimization, operations research, information theory, nonlinear analysis, machine learning and more. This paper re-examines various aspects related to the theory of Bregman functions and divergences. In particular, it presents many sufficient conditions which allow the construction of Bregman functions in a general setting and introduces new Bregman functions (such as a negative iterated log entropy). Moreover, it sheds new light on several known Bregman functions such as quadratic entropies, the negative Havrda-Charvát-Tsallis entropy, and the negative Boltzmann-Gibbs-Shannon entropy, and it shows that the negative Burg entropy, which is not a Bregman function according to the classical theory but nevertheless is known to have "Bregmanian properties", can, by our re-examination of the theory, be considered as a Bregman function. Our analysis yields several by-products of independent interest such as the introduction of the concept of relative uniform convexity (a certain generalization of uniform convexity), new properties of uniformly and strongly convex functions, and results in Banach space theory.

A Telescopic Bregmanian Proximal Gradient Method Without the Global Lipschitz Continuity Assumption

“…Since r ≥ 2, it follows that x 2 ≥ x r for every x ∈ R n . This inequality and simple calculations show that b is strongly convex on (X, · r ), again with 1 as a strong convexity parameter (for a more general statement, see [40,Proposition 5.3]). Thus b is strongly convex on S k (for each k ∈ N) with µ k := 1 as a strong convexity parameter.…”

“…k=1 is increasing and Assumption 5.3 holds. Finally, [40,Subsection 6.4] shows that for each x ∈ U (in particular, for each x ∈ C), there exists r x ≥ 0 such that b is uniformly convex relative to ({x}, {y ∈ U : x ≥ r x }), with some gauge ψ x satisfying lim t→∞ ψ x (t) = ∞ (namely, r x := 2 x and ψ x (t) = γt, t ∈ [0, ∞[, for some γ > 0 independent of t). Thus, if we denote τ k := L k for every k ∈ N, then Theorem 6.1 and Remark 5.8 imply that the proximal sequence (x k ) ∞ k=1 generated by Algorithm 5.5 converges to a point in MIN(F ), and (19) implies that the non-asymptotic rate of convergence is O(1/k 0.25 ).…”

“…A major advantage of the above-mentioned dependence of the iterative steps on well-chosen subsets is that it gives the users a lot of flexibility and, in particular, allows them to assume that the gradient of the smooth term is Lipschitz continuous merely on the above-mentioned subsets (an assumption which frequently holds since these subsets are often bounded), rather than globally Lipschitz continuous. Based on results which have recently been established in [40], we obtain, under certain practical conditions, explicit non-asymptotic rates of convergence to the optimal value (that is, convergence of the function values to the optimal value), as well as the weak convergence of the whole sequence to a minimizer. In fact, we show that sometimes sublinear non-asymptotic rates of convergence can be obtained, and sometimes these non-asymptotic rates can be arbitrarily close to the sublinear rate.…”

“…This divergence is not a true metric (for example, because it does not satisfy the triangle inequality), but it still enjoys various properties which make it a useful substitute for a distance (for example, B(x, y) ≥ 0 for all x ∈ dom(b) and y ∈ Int(dom(b)) and B(x, y) = 0 if and only if x = y ∈ U : see, for instance, [40, Proposition 4.13(III)]). Many more details about this notion, including historical details, a long list of relevant references, various mathematical properties, and a thorough re-examination, can be found in the recent paper [40]. (ii) A semi-Bregman function generalizes the notion of "a Bregman function" (the term "semi-Bregman function" seems to be new, although it has been used here and there without this explicit name Here we briefly recall very well-known versions of the proximal gradient method (the proximal point algorithm).…”

“…It is well known and can easily be proved that b is strongly convex on R n with the Euclidean norm, where the strong convexity parameter is 1: see, for instance, [40,Subsection 11.4]. Since r ≥ 2, it follows that x 2 ≥ x r for every x ∈ R n .…”