Bridging Maximum Likelihood and Adversarial Learning via α-Divergence

Abstract

Maximum likelihood (ML) and adversarial learning are two popular approaches for training generative models, and from many perspectives these techniques are complementary. ML learning encourages the capture of all data modes, and it is typically characterized by stable training. However, ML learning tends to distribute probability mass diffusely over the data space, e.g., yielding blurry synthetic images. Adversarial learning is well known to synthesize highly realistic natural images, despite practical challenges like mode dropping and delicate training. We propose an α-Bridge to unify the advantages of ML and adversarial learning, enabling the smooth transfer from one to the other via the α-divergence. We reveal that generalizations of the α-Bridge are closely related to approaches developed recently to regularize adversarial learning, providing insights into that prior work, and further understanding of why the α-Bridge performs well in practice.

Using Probabilistic Machine Learning to Better Model Temporal Patterns in Parameterizations: a case study with the Lorenz 96 model

“…Poor performance in certain aspects may be overshadowed by good performance elsewhere. This can result in cases where increased hold-out likelihoods do not correspond to better sample quality, as noted in the ML literature (Goodfellow et al, 2014;Grover et al, 2018;Theis et al, 2015;Zhao et al, 2020) and seen in our results: the RNN has a worse F = 28 hold-out likelihood than the polynomial, yet the polynomial model explodes unlike the RNN. In this case, the phenomenon of 'explosion' is not significantly penalising the likelihood.…”

Using Probabilistic Machine Learning to Better Model Temporal Patterns in Parameterizations: a case study with the Lorenz 96 model

“…Poor performance in certain aspects may be overshadowed by good performance elsewhere. This can result in cases where increased hold-out likelihoods do not always correspond to better sample quality, as noted in the ML literature (Goodfellow et al, 2014;Grover et al, 2018;Theis et al, 2015;Zhao et al, 2020). In our work, although the RNN has the greatest likelihood for F = 23, it does not have the smallest KL-divergence for the 1D histograms in Figure 7.…”

Least kth-Order and Rényi Generative Adversarial Networks

“…The use of information theory to study and improve neural networks is a relatively new yet promising direction of research (Lee, Tran, & Cheung, 2021;Nowozin, Cseke, & Tomioka, 2016;Principe, 2010;Achille & Soatto, 2019;Chen et al, 2016;Wickstrom et al, 2019;Tishby & Zaslavsky, 2015;Alemi, Fischer, Dillon, & Murphy, 2017;Zhao, Cong, Dai, & Carin, 2020;Zaidi, Estella-Aguerri, & Shitz, 2020). While many GANs loss functions are based on the Jensen-Shannon divergence, there are other divergence measures and tools in information theory that can be directly applied to the design of GANs.…”

“…The family of loss functions that simplify down to f -divergences was thoroughly studied in Nowozin et al (2016), Farnia & Tse (2018), and Li et al (2019). Bridging the gap between maximum likelihood learning and GANs, especially those with loss functions that simplify down to f -divergences, has also been analyzed in Zhao et al (2020). Using the symmetric Kullback-Leibler (KL) divergence, researchers have also shown that a variant of VAEs is connected to GANs (Chen et al, 2018).…”