Gotta Go Fast When Generating Data with Score-Based Models

Abstract

Score-based (denoising diffusion) generative models have recently gained a lot of success in generating realistic and diverse data. These approaches define a forward diffusion process for transforming data to noise and generate data by reversing it (thereby going from noise to data). Unfortunately, current score-based models generate data very slowly due to the sheer number of score network evaluations required by numerical SDE solvers. In this work, we aim to accelerate this process by devising a more efficient SDE solver. Existing approaches rely on the Euler-Maruyama (EM) solver, which uses a fixed step size. We found that naively replacing it with other SDE solvers fares poorly -they either result in low-quality samples or become slower than EM. To get around this issue, we carefully devise an SDE solver with adaptive step sizes tailored to score-based generative models piece by piece. Our solver requires only two score function evaluations, rarely rejects samples, and leads to high-quality samples. Our approach generates data 2 to 10 times faster than EM while achieving better or equal sample quality. For high-resolution images, our method leads to significantly higher quality samples than all other methods tested. Our SDE solver has the benefit of requiring no step size tuning. Code is available on https://github.com/AlexiaJM/score_sde_fast_sampling. † Equal contribution Preprint. Under review.

Score-Based Generative Modeling with Critically-Damped Langevin Diffusion

“…( 2) in the main text. This can be achieved, for example, via our novel SSCS, Euler-Maruyama, or methods such as GGF (Jolicoeur-Martineau et al, 2021a). However, Song et al (2021b;c) have shown that a corresponding ordinary differential equation can be defined that generates samples from the same distribution, in case s θ (u t , t) models the ground truth scores perfectly.…”

“…B.5). To simulate the SDE of the reverse-time diffusion process, previous works often relied on Euler-Maruyama (EM) (Kloeden & Platen, 1992) and related methods (Ho et al, 2020;Song et al, 2021c;Jolicoeur-Martineau et al, 2021a). We derive a new solver, tailored to CLD-based models.…”

“…A complementary work to CLD is "Gotta Go Fast" (GGF) (Jolicoeur-Martineau et al, 2021a), which introduces an adaptive SDE solver for SGMs, tuned towards image synthesis. GGF uses standard Euler-based methods under the hood (Kloeden & Platen, 1992;Roberts, 2012), in contrast to our SSCS that is derived from first principles.…”

“…Note that in practice we do not use SSCS to integrate all the way from t = 0 to t = T , but only up to t = T − , and perform a denoising step, similar to previous works (Jolicoeur-Martineau et al, 2021a;Song et al, 2021c). It is worth noting that our SSCS scheme would also be applicable when we used time-dependent β(t), as in our more general derivation of the CLD perturbation kernel in App.…”

Tackling the Generative Learning Trilemma with Denoising Diffusion GANs

“…One major drawback of diffusion or score-based models is the slow sampling speed due to a large number of iterative sampling steps. To alleviate this issue, multiple methods have been proposed, including knowledge distillation (Luhman & Luhman, 2021), learning an adaptive noise schedule (San-Roman et al, 2021), introducing non-Markovian diffusion processes , and using better SDE solvers for continuous-time models (Jolicoeur-Martineau et al, 2021a). In particular, Song et al (2021a) uses x 0 sampling as a crucial ingredient to their method, but their denoising distribution is still a Gaussian.…”