Distribution Matching in Variational Inference

Roșca, Mihaela; Lakshminarayanan, Balaji; Mohamed, Shakir

doi:10.48550/arxiv.1802.06847

Cited by 24 publications

(50 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As discussed in Section 2.1, issue i) is compromising the VAE framework in any case, as reported in several works (Hoffman & Johnson, 2016;Rosca et al, 2018;Dai & Wipf, 2019). To fix this, some works extend the VAE objective by encouraging the aggregated posterior to match p(z) (Tolstikhin et al, 2017) or by utilizing more complex priors (Kingma et al, 2016;Tomczak & Welling, 2018;Bauer & Mnih, 2019).…”

Section: Ex-post Density Estimationmentioning

confidence: 83%

From Variational to Deterministic Autoencoders

Ghosh,

Sajjadi,

Vergari

et al. 2019

Preprint

View full text Add to dashboard Cite

Variational Autoencoders (VAEs) provide a theoretically-backed and popular framework for deep generative models. However, learning a VAE from data poses still unanswered theoretical questions and considerable practical challenges. In this work, we propose an alternative framework for generative modeling that is simpler, easier to train, and deterministic, yet has many of the advantages of the VAE. We observe that sampling a stochastic encoder in a Gaussian VAE can be interpreted as simply injecting noise into the input of a deterministic decoder. We investigate how substituting this kind of stochasticity, with other explicit and implicit regularization schemes, can lead to an equally smooth and meaningful latent space without having to force it to conform to an arbitrarily chosen prior. To retrieve a generative mechanism to sample new data points, we introduce an ex-post density estimation step that can be readily applied to the proposed framework as well as existing VAEs, improving their sample quality. We show, in a rigorous empirical study, that the proposed regularized deterministic autoencoders are able to generate samples that are comparable to, or better than, those of VAEs and more powerful alternatives when applied to images as well as to structured data such as molecules.

show abstract

Section: Ex-post Density Estimationmentioning

confidence: 83%

From Variational to Deterministic Autoencoders

Ghosh,

Sajjadi,

Vergari

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…While D2C is a special case of VAE, we argue that D2C is non-trivial in the sense that it addresses a long-standing problem in VAE methods [87,84], namely the mismatch between the prior distribution p θ (z) and the aggregate (approximate) posterior distribution q φ (z) := E p data (x) [q φ (z|x)]. A mismatch could create "holes" [76,41,3] in the prior that the aggregate posterior fails to cover during training, resulting in worse sample quality, as many latent variables used during generation are likely to never have been trained on. We formalize this notion in the following definition.…”

Section: D2c Models Address Latent Posterior Mismatch In Vaesmentioning

confidence: 99%

D2C: Diffusion-Denoising Models for Few-shot Conditional Generation

Sinha¹,

Song²,

Meng³

et al. 2021

Preprint

View full text Add to dashboard Cite

Conditional generative models of high-dimensional images have many applications, but supervision signals from conditions to images can be expensive to acquire. This paper describes Diffusion-Decoding models with Contrastive representations (D2C), a paradigm for training unconditional variational autoencoders (VAEs) for few-shot conditional image generation. D2C uses a learned diffusion-based prior over the latent representations to improve generation and contrastive selfsupervised learning to improve representation quality. D2C can adapt to novel generation tasks conditioned on labels or manipulation constraints, by learning from as few as 100 labeled examples. On conditional generation from new labels, D2C achieves superior performance over state-of-the-art VAEs and diffusion models. On conditional image manipulation, D2C generations are two orders of magnitude faster to produce over StyleGAN2 ones and are preferred by 50% − 60% of the human evaluators in a double-blind study. * Equal contribution.Preprint. Under review.

show abstract

“…So, to sample from our autoencoder, we need an additional mechanism to sample z sem ∈ R d from the latent distribution. While VAE is an appealing choice for this task, balancing between retaining rich information in the latent code and maintaining the sampling quality in VAE is hard [27,32,33,35]. GAN is another choice, though it complicates training stability, which is one main strength of DPMs.…”

Section: Sampling With Diffusion Autoencodersmentioning

confidence: 99%