Implicit Generation and Generalization in Energy-Based Models

Du, Yu; Mordatch, Igor

doi:10.48550/arxiv.1903.08689

Cited by 73 publications

(178 citation statements)

References 21 publications

Supporting

Mentioning

157

Contrasting

Order By: Relevance

“…To put this goal in context, in recent energy models for high-dimensional data (Xie et al, 2016;Nijkamp et al, 2019;Du & Mordatch, 2019;Zhao et al, 2020;Du et al, 2020;Xie et al, 2021), sampling using MCMC quickly breaks or collapses to a mode and chains longer than a few hundred steps were not reported. Thus, evaluation in prior work relies on samples from independent MCMC chains, where in addition there are heuristics like "replay buffer" (Du & Mordatch, 2019), not supported by theory. In this work, we report FID scores obtained by single MCMC chains, the first result of its kind, which we consider as a benchmark for future works on long run MCMC chains.…”

Section: Resultsmentioning

confidence: 99%

Multimeasurement Generative Models

Saremi¹,

Srivastava²

2021

Preprint

View full text Add to dashboard Cite

We formally map the problem of sampling from an unknown distribution with density p X in R d to the problem of learning and sampling p Y in R M d obtained by convolving p X with a fixed factorial kernel: p Y is referred to as M-density and the factorial kernel as multimeasurement noise model (MNM). The M-density is smoother than p X , easier to learn and sample from, yet for large M the two problems are mathematically equivalent since X can be estimated exactly given Y = y using the Bayes estimator x(y) = E[X|Y = y]. To formulate the problem, we derive x(y) for Poisson and Gaussian MNMs expressed in closed form in terms of unnormalized p Y . This leads to a simple least-squares objective for learning parametric energy and score functions. We present various parametrization schemes of interest, including one in which studying Gaussian M-densities directly leads to multidenoising autoencoders-this is the first theoretical connection made between denoising autoencoders and empirical Bayes in the literature. Samples from p X are obtained by walk-jump sampling (Saremi & Hyvärinen, 2019) via underdamped Langevin MCMC (walk) to sample from p Y and the multimeasurement Bayes estimation of X (jump). We study permutation invariant Gaussian M-densities on MNIST, CIFAR-10, and FFHQ-256 datasets, and demonstrate the effectiveness of this framework for realizing fast-mixing stable Markov chains in high dimensions. INTRODUCTIONConsider a collection of i.i.d. samples {x i } n i=1 , assumed to have been drawn from an unknown distribution with density p X in R d . An important problem in probabilistic modeling is the task of drawing independent samples from p X , which has numerous potential applications. This problem is typically approached in two phases: approximating p X , and drawing samples from the approximated density. In unnormalized models the first phase is approached by learning the energy function f X associated with the Gibbs distribution p X ∝ exp(−f X ), and for the second phase one must resort to Markov chain Monte Carlo methods, such as Langevin MCMC, which are typically very slow to mix in high dimensions. MCMC sampling is considered an "art" and we do not have black box samplers that converge fast and are stable for complex (natural) distributions. The source of the problem is mainly attributed to the fact that the energy functions of interest are typically highly nonconvex.A broad sketch of our solution to this problem is to model a smoother density in an M-fold expanded space. The new density p(y), called M-density, is defined in R M d , where the bold y is a shorthand for (y 1 , . . . , y M ). M-density is smoother in the sense that its marginals p m (y m ) are obtained by the convolution p m (y m ) = p m (y m |x)p(x)dx with a smoothing kernel p m (y m |x) which for most of the paper we take to be the isotropic Gaussian:Although we bypass learning p(x), the new formalism allows for generating samples from p(x) since X can be estimated exactly given Y = y (for large M ). To give a physical picture, the approac...

show abstract

Section: Resultsmentioning

confidence: 99%

Multimeasurement Generative Models

Saremi¹,

Srivastava²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Energy Based Models. Our work is related to existing work on energy-based models [5,7,9,11,13,23,34,43,46]. Most similar to our work is that of [8], which proposes a framework of utilizing EBMs to compose several object descriptions together.…”

Section: Generated Imagementioning

confidence: 90%

“…We model each relational probability distribution utilizing an Energy-Based Model (EBM) [7,27]. EBMs are a class of unnormalized probability models, which parameterize a probability distribution p θ (x) utilizing a learned energy function E θ (x):…”

Section: Energy-based Modelsmentioning

confidence: 99%

Learning to Compose Visual Relations

Liu¹,

Li²,

Du³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

The visual world around us can be described as a structured set of objects and their associated relations. An image of a room may be conjured given only the description of the underlying objects and their associated relations. While there has been significant work on designing deep neural networks which may compose individual objects together, less work has been done on composing the individual relations between objects. A principal difficulty is that while the placement of objects is mutually independent, their relations are entangled and dependent on each other. To circumvent this issue, existing works primarily compose relations by utilizing a holistic encoder, in the form of text or graphs. In this work, we instead propose to represent each relation as an unnormalized density (an energy-based model), enabling us to compose separate relations in a factorized manner. We show that such a factorized decomposition allows the model to both generate and edit scenes that have multiple sets of relations more faithfully. We further show that decomposition enables our model to effectively understand the underlying relational scene structure. Project page at: https://composevisualrelations.github.io/ * indicates equal contribution 35th Conference on Neural Information Processing Systems (NeurIPS 2021).

show abstract

“…(Ho et al, 2020) 6.12 90 2.64 DDIM, K = 30 (Song et al, 2020a) 5.85 30 0.88 Improved DDPM, K = 45 5.96 45 1.37 SNGAN (Miyato et al, 2018) 21.7 1 -BigGAN (cond.) (Brock et al, 2018) 14.73 1 -StyleGAN2 (Karras et al, 2020a) 8.32 1 -StyleGAN2 + ADA (Karras et al, 2020a) 2.92 1 -NVAE 23.5 1 -Glow (Kingma & Dhariwal, 2018) 48.9 1 -EBM (Du & Mordatch, 2019) 38.2 60 -VAEBM (Xiao et al, 2020) 12.2 16 -G.9 SAMPLES…”

Section: G8 Comparison To Other Classes Of Generative Modelsmentioning

confidence: 99%

Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models

Bao¹,

Li²,

Zhu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Diffusion probabilistic models (DPMs) represent a class of powerful generative models. Despite their success, the inference of DPMs is expensive since it generally needs to iterate over thousands of timesteps. A key problem in the inference is to estimate the variance in each timestep of the reverse process. In this work, we present a surprising result that both the optimal reverse variance and the corresponding optimal KL divergence of a DPM have analytic forms w.r.t. its score function. Building upon it, we propose Analytic-DPM, a training-free inference framework that estimates the analytic forms of the variance and KL divergence using the Monte Carlo method and a pretrained score-based model. Further, to correct the potential bias caused by the score-based model, we derive both lower and upper bounds of the optimal variance and clip the estimate for a better result. Empirically, our analytic-DPM improves the log-likelihood of various DPMs, produces high-quality samples, and meanwhile enjoys a 20× to 80× speed up.

show abstract

Implicit Generation and Generalization in Energy-Based Models

Cited by 73 publications

References 21 publications

Multimeasurement Generative Models

Multimeasurement Generative Models

Learning to Compose Visual Relations

Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models

Contact Info

Product

Resources

About