Accelerating Diffusion Models via Early Stop of the Diffusion Process

Abstract

Denoising Diffusion Probabilistic Models (DDPMs) have achieved impressive performance on various generation tasks. By modeling the reverse process of gradually diffusing the data distribution into a Gaussian distribution, generating a sample in DDPMs can be regarded as iteratively denoising a randomly sampled Gaussian noise. However, in practice DDPMs often need hundreds or even thousands of denoising steps to obtain a high-quality sample from the Gaussian noise, leading to extremely low inference efficiency. In this work, we propose a principled acceleration strategy, referred to as Early-Stopped DDPM (ES-DDPM), for DDPMs. The key idea is to stop the diffusion process early where only the few initial diffusing steps are considered and the reverse denoising process starts from a non-Gaussian distribution. By further adopting a powerful pre-trained generative model, such as GAN and VAE, in ES-DDPM, sampling from the target non-Gaussian distribution can be efficiently achieved by diffusing samples obtained from the pre-trained generative model. In this way, the number of required denoising steps is significantly reduced. In the meantime, the sample quality of ES-DDPM also improves substantially, outperforming both the vanilla DDPM and the adopted pre-trained generative model. On extensive experiments across CIFAR-10, CelebA, ImageNet, LSUN-Bedroom and LSUN-Cat, ES-DDPM obtains promising acceleration effect and performance improvement over representative baseline methods. Moreover, ES-DDPM also demonstrates several attractive properties, including being orthogonal to existing acceleration methods, as well as simultaneously enabling both global semantic and local pixel-level control in image generation.Preprint. Under review.

Accelerating Diffusion Models via Pre-segmentation Diffusion Sampling for Medical Image Segmentation

“…Vanilla DDPM [5] 0.796 4.179 0.669 0.779 TDPM [12] 0.801 3.608 0.676 0.776 ES-DDPM [13] 0.803 3.523 0.678 0.777 PD-DDPM 0.812 3.494 0.689 0.800…”

“…Bayesian U-Net [20] and Probabilistic U-Net [21] are representative methods that can estimate the uncertainty of segmentation. And we also compare PD-DDPM with other accelerating DDPMs, including TDPM [12] and ES-DDPM [13]. It should be emphasized that the size of ensemble in the comparison methods is also set to 5.…”

“…( | ) of power generation. Some other methods [12,13] improve sampling efficiency by truncating the forward and reverse diffusion processes, boosting the performance at the same time. But this method needs to combine GAN [14] or VAE [15] models that are difficult to train.…”

ERA-Solver: Error-Robust Adams Solver for Fast Sampling of Diffusion Probabilistic Models

“…It usually requires hundreds or even thousands of network evaluations, which limits various downstream applications for DDPMs. There already existed methods (Watson et al, 2021;Lyu et al, 2022;Lam et al, 2022;Salimans & Ho, 2021) which depend on extra training stage to derive fast sampling methods. The trainingbased methods usually require tremendous training costs for different data manifolds and tasks, which inspires many works to explore a training-free sampler based on numerical methods.…”

To the Noise and Back: Diffusion for Shared Autonomy

“…Although we independently came up with this idea, a others have explored a similar approach for image editing [33]. Related, many recent papers have consider running reverse diffusion from some intermediate step rather than pure Gaussian noise [32,50,38,52], as we do in this paper.…”