On Jensen–Rényi and Jeffreys–Rényi type <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline" id="d1e22" altimg="si4.svg"><mml:mi>f</mml:mi></mml:math>-divergences induced by convex functions

Kluza, Paweł A.

doi:10.1016/j.physa.2019.122527

Cited by 9 publications

(7 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Rényi used the simplest set of postulates that characterize Shannon's entropy and introduced his own entropy and divergence measures (parameterized by its order α) that generalize the Shannon entropy and the KL divergence, respectively (Rényi, 1961). Moreover, the original Jensen-Rényi divergence (He, Hamza, & Krim, 2003) as well as the identically named divergence (Kluza, 2019) used in this letter are non-f -divergence generalizations of the Jensen-Shannon divergence. Traditionally, Rényi's entropy and divergence have had applications in a wide range of problems, including lossless data compression (Campbell, 1965;Courtade & Verdú, 2014;Rached, Alajaji, & Campbell, 1999), hypothesis testing (Csiszár, 1995;Alajaji, Chen, & Rached, 2004), error probability (Ben-Bassat & Raviv, 2006), and guessing (Arikan, 1996;Verdú, 2015).…”

Section: Prior Workmentioning

confidence: 99%

“…With these Rényi measures in place, we propose a new GAN's generator loss function expressed in terms of the negative sum of two Rényi crossentropy functionals. We show that minimizing this α-parameterized loss function under an optimal discriminator results in the minimization of the Jensen-Rényi divergence (Kluza, 2019), which is a natural extension of the Jensen-Shannon divergence as it uses the Rényi divergence instead of the Kullback-Leibler (KL) divergence in its expression. 1 We also prove that our generator loss function of order α converges to the original GAN loss function in Goodfellow et al (2014) when α → 1.…”

Section: Contributionsmentioning

confidence: 99%

See 1 more Smart Citation

Least kth-Order and Rényi Generative Adversarial Networks

et al. 2021

View full text Add to dashboard Cite

We investigate the use of parameterized families of information-theoretic measures to generalize the loss functions of generative adversarial networks (GANs) with the objective of improving performance. A new generator loss function, least kth-order GAN (LkGAN), is introduced, generalizing the least squares GANs (LSGANs) by using a kth-order absolute error distortion measure with k≥1 (which recovers the LSGAN loss function when k=2). It is shown that minimizing this generalized loss function under an (unconstrained) optimal discriminator is equivalent to minimizing the kth-order Pearson-Vajda divergence. Another novel GAN generator loss function is next proposed in terms of Rényi cross-entropy functionals with order α>0, α≠1. It is demonstrated that this Rényi-centric generalized loss function, which provably reduces to the original GAN loss function as α→1, preserves the equilibrium point satisfied by the original GAN based on the Jensen-Rényi divergence, a natural extension of the Jensen-Shannon divergence. Experimental results indicate that the proposed loss functions, applied to the MNIST and CelebA data sets, under both DCGAN and StyleGAN architectures, confer performance benefits by virtue of the extra degrees of freedom provided by the parameters k and α, respectively. More specifically, experiments show improvements with regard to the quality of the generated images as measured by the Fréchet inception distance score and training stability. While it was applied to GANs in this study, the proposed approach is generic and can be used in other applications of information theory to deep learning, for example, the issues of fairness or privacy in artificial intelligence.

show abstract

Section: Prior Workmentioning

confidence: 99%

Section: Contributionsmentioning

confidence: 99%

Least kth-Order and Rényi Generative Adversarial Networks

et al. 2021

View full text Add to dashboard Cite

show abstract

“…If, additionally, α tends to 1 then based on the proof of the Equation ( 11 ) from [ 16 ], Sharma–Mittal h-divergence tends to relative entropy (called Kullback–Leibler divergence).…”

Section: Sharma–mittal Type Divergencesmentioning

confidence: 99%

“…When in ( 8 ) and ( 9 )

and we substitute for

then we obtain, defined in [ 16 ], the generalized

Jensen–Rényi and Jeffreys–Rényi divergences, respectively:

…”

Section: Jensen–sharma–mittal and Jeffreys–sharma–mittal Divergencesmentioning

confidence: 99%

See 1 more Smart Citation

Inequalities for Jensen–Sharma–Mittal and Jeffreys–Sharma–Mittal Type f–Divergences

Kluza

2021

Entropy

Self Cite

View full text Add to dashboard Cite

In this paper, we introduce new divergences called Jensen–Sharma–Mittal and Jeffreys–Sharma–Mittal in relation to convex functions. Some theorems, which give the lower and upper bounds for two new introduced divergences, are provided. The obtained results imply some new inequalities corresponding to known divergences. Some examples, which show that these are the generalizations of Rényi, Tsallis, and Kullback–Leibler types of divergences, are provided in order to show a few applications of new divergences.

show abstract