Denoising Diffusion Implicit Models

Abstract

Denoising diffusion probabilistic models (DDPMs) have achieved high quality image generation without adversarial training, yet they require simulating a Markov chain for many steps to produce a sample. To accelerate sampling, we present denoising diffusion implicit models (DDIMs), a more efficient class of iterative implicit probabilistic models with the same training procedure as DDPMs. In DDPMs, the generative process is defined as the reverse of a Markovian diffusion process. We construct a class of non-Markovian diffusion processes that lead to the same training objective, but whose reverse process can be much faster to sample from. We empirically demonstrate that DDIMs can produce high quality samples 10× to 50× faster in terms of wall-clock time compared to DDPMs, allow us to trade off computation for sample quality, and can perform semantically meaningful image interpolation directly in the latent space. Our implementation is available at this link.

Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models

“…Empirically, Analytic-DPM consistently improves the log-likelihood of these DPMs and meanwhile enjoys a 20× to 40× speed up. Besides, Analytic-DPM also consistently improves the sample quality of DDIMs (Song et al, 2020a) and requires up to 50 timesteps (which is a 20× to 80× speed up compared to the full timesteps) to achieve a comparable FID to the corresponding baseline.…”

“…Despite their success, the inference of DPMs (e.g., sampling and density evaluation) often requires to iterate over thousands of timesteps, which is two or three orders of magnitude slower (Song et al, 2020a) than other generative models such as GANs. A key problem in the inference is to estimate the variance in each timestep of the reverse process.…”

“…Analytic-DPM is applicable to a variety of DPMs (Ho et al, 2020;Song et al, 2020a; in a plug-and-play manner. Empirically, Analytic-DPM consistently improves the log-likelihood of these DPMs and meanwhile enjoys a 20× to 40× speed up.…”

“…where I is the identity matrix, α n and β n are scalars and α n := 1−β n . Song et al (2020a) introduce a more general non-Markov process indexed by a non-negative vector λ = (λ 1 , • • • , λ N ) ∈ R N ≥0 :…”

“…In DDPMs (Ho et al, 2020), two commonly used settings are σ 2 n = β n and σ 2 n = βn . In DDIMs, Song et al (2020a) consistently use σ 2 n = λ 2 n . We argue that these handcrafted values are not the true optimal solution of Eq.…”