A Variational Perspective on Diffusion-Based Generative Models and Score Matching

Huang, Chin-Wei; Lim, Jae Hyun; Courville, Aaron

doi:10.48550/arxiv.2106.02808

Cited by 6 publications

(14 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Concurrent to our work, Song et al [2021a], Huang et al [2021], and Vahdat et al [2021] also derived variational lower bounds to the data likelihood under a continuous-time diffusion model. Where we consider the infinitely deep limit of a standard VAE, Song et al [2021a] and Vahdat et al [2021] present different derivations based on stochastic differential equations.…”

Section: Related Worksupporting

confidence: 65%

“…Where we consider the infinitely deep limit of a standard VAE, Song et al [2021a] and Vahdat et al [2021] present different derivations based on stochastic differential equations. Huang et al [2021] considers both perspectives and discusses the similarities between the two approaches. An advantage of our analysis compared to these other works is that we present an intuitive expression of the VLB in terms of the signal-to-noise ratio of the diffused data, which then leads to new results on the invariance of the generative model and its VLB to the specification of the diffusion process.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Variational Diffusion Models

Kingma¹,

Salimans²,

Poole³

et al. 2021

Preprint

View full text Add to dashboard Cite

Diffusion-based generative models have demonstrated a capacity for perceptually impressive synthesis, but can they also be great likelihood-based models? We answer this in the affirmative, and introduce a family of diffusion-based generative models that obtain state-of-the-art likelihoods on standard image density estimation benchmarks. Unlike other diffusion-based models, our method allows for efficient optimization of the noise schedule jointly with the rest of the model. We show that the variational lower bound (VLB) simplifies to a remarkably short expression in terms of the signal-to-noise ratio of the diffused data, thereby improving our theoretical understanding of this model class. Using this insight, we prove an equivalence between several models proposed in the literature. In addition, we show that the continuous-time VLB is invariant to the noise schedule, except for the signal-to-noise ratio at its endpoints. This enables us to learn a noise schedule that minimizes the variance of the resulting VLB estimator, leading to faster optimization. Combining these advances with architectural improvements, we obtain state-of-the-art likelihoods on image density estimation benchmarks, outperforming autoregressive models that have dominated these benchmarks for many years, with often significantly faster optimization. In addition, we show how to turn the model into a bits-back compression scheme, and demonstrate lossless compression rates close to the theoretical optimum. * Equal contribution.Preprint. Under review.

show abstract

Section: Related Worksupporting

confidence: 65%

Section: Related Workmentioning

confidence: 99%

Variational Diffusion Models

Kingma¹,

Salimans²,

Poole³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Diffusion models are capable of generating high-quality images that can compete and even outperform the latest GAN methods [10,16,33,40]. A variational framework for the likelihood estimation of diffusion models is introduced by Huang et al [19]. Subsequently, Kingma et al [21] proposed a Variational Diffusion Model that produces state-of-the-art results in likelihood estimation for image density.…”

Section: Related Workmentioning

confidence: 99%

SegDiff: Image Segmentation with Diffusion Probabilistic Models

Amit¹,

Nachmani²,

Shaharbany³

et al. 2021

Preprint

View full text Add to dashboard Cite

Diffusion Probabilistic Methods are employed for stateof-the-art image generation. In this work, we present a method for extending such models for performing image segmentation. The method learns end-to-end, without relying on a pre-trained backbone. The information in the input image and in the current estimation of the segmentation map is merged by summing the output of two encoders. Additional encoding layers and a decoder are then used to iteratively refine the segmentation map using a diffusion model. Since the diffusion model is probabilistic, it is applied multiple times and the results are merged into a final segmentation map. The new method obtains state-of-the-art results on the Cityscapes validation set, the Vaihingen building segmentation benchmark, and the MoNuSeg dataset.

show abstract

“…In practice, the choice of λ(t) can largely affect the performance of SGM. Fortunately, recent works (Song et al, 2021;Huang et al, 2021) have shown that the log-likelihood of SGM, despite being complex, can be parameterized as follows:…”

Section: Score-based Generative Model (Sgm)mentioning

confidence: 99%

“…The underlying relation between the optimization principle of SB and modern generative training, in particular SGM, remains relatively unexplored, despite their intimately related problem formulations. More importantly, with the recent connection between SGM and log-likelihood estimation (Song et al, 2021;Huang et al, 2021), it is crucial to explore whether there exists an alternative way of training SB that better respects, or perhaps generalizes, modern training of SGM, so as to solidify the suitability of SB as a principled generative model.…”

Section: Introductionmentioning

confidence: 99%

Likelihood Training of Schrödinger Bridge using Forward-Backward SDEs Theory

Chen¹,

Liu²,

Theodorou³

2021

Preprint

View full text Add to dashboard Cite

Schrödinger Bridge (SB) is an optimal transport problem that has received increasing attention in deep generative modeling for its mathematical flexibility compared to the Scored-based Generative Model (SGM). However, it remains unclear whether the optimization principle of SB relates to the modern training of deep generative models, which often rely on constructing parameterized loglikelihood objectives.This raises questions on the suitability of SB models as a principled alternative for generative applications. In this work, we present a novel computational framework for likelihood training of SB models grounded on Forward-Backward Stochastic Differential Equations Theory -a mathematical methodology appeared in stochastic optimal control that transforms the optimality condition of SB into a set of SDEs. Crucially, these SDEs can be used to construct the likelihood objectives for SB that, surprisingly, generalizes the ones for SGM as special cases. This leads to a new optimization principle that inherits the same SB optimality yet without losing applications of modern generative training techniques, and we show that the resulting training algorithm achieves comparable results on generating realistic images on MNIST, CelebA, and CIFAR10. * Equal contribution. Order determined by coin flip. See Author Contributions section.

show abstract

A Variational Perspective on Diffusion-Based Generative Models and Score Matching

Cited by 6 publications

References 35 publications

Variational Diffusion Models

Variational Diffusion Models

SegDiff: Image Segmentation with Diffusion Probabilistic Models

Likelihood Training of Schrödinger Bridge using Forward-Backward SDEs Theory

Contact Info

Product

Resources

About