Score-Based Generative Modeling with Critically-Damped Langevin Diffusion

Dockhorn, Tim; Vahdat, Arash; Kreis, Karsten

doi:10.48550/arxiv.2112.07068

Cited by 16 publications

(46 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To the best of our knowledge, we are the first to propose a diffusion process for graphs and further model the dependency through a system of SDEs. It is notable that recent developments of score-based generative methods, such as latent score-based generative model (LSGM) (Vahdat et al, 2021) and critically-damped Langevin diffusion (CLD) (Dockhorn et al, 2021), are complementary to our method as we can apply these methods to improve each component-wise diffusion process.…”

Section: Related Workmentioning

confidence: 99%

“…However, solving the system of two diffusion processes that are tied by the partial scores brings about another difficulty. Thus we propose a novel integrator, Symmetric Splitting for System of SDEs (S4) that is efficient yet accurate, inspired by the Symmetric Splitting CLD Sampler (SSCS) (Dockhorn et al, 2021) and Predictor-Corrector Sampler (PC sampler) (Song et al, 2021b). Specifically, at each discretized time step t, S4 consists of three steps: the score computation, the prediction, and the correction.…”

Section: Solving the System Of Reverse-time Sdesmentioning

confidence: 99%

“…F on sufficiently small half-step δt/2. By using the Euler method, the prediction step has the same order of error as EM with local error O(δt 2 ) and global error O(δt) (Dockhorn et al, 2021). We provide the pseudo-code for the S4 solver in Algorithm 1 of Appendix.…”

Section: *mentioning

confidence: 99%

“…F and e δt L * S with the Trotter theorem (Trotter, 1959), we can approximate the intractable solution e t( L * F + L * S ) as follows (Dockhorn et al, 2021):…”

Section: *mentioning

confidence: 99%

See 3 more Smart Citations

Score-based Generative Modeling of Graphs via the System of Stochastic Differential Equations

Jo¹,

Lee²,

Hwang³

2022

Preprint

View full text Add to dashboard Cite

Generating graph-structured data requires learning the underlying distribution of graphs. Yet, this is a challenging problem, and the previous graph generative methods either fail to capture the permutation-invariance property of graphs or cannot sufficiently model the complex dependency between nodes and edges, which is crucial for generating real-world graphs such as molecules. To overcome such limitations, we propose a novel score-based generative model for graphs with a continuous-time framework. Specifically, we propose a new graph diffusion process that models the joint distribution of the nodes and edges through a system of stochastic differential equations (SDEs). Then, we derive novel score matching objectives tailored for the proposed diffusion process to estimate the gradient of the joint logdensity with respect to each component, and introduce a new solver for the system of SDEs to efficiently sample from the reverse diffusion process. We validate our graph generation method on diverse datasets, on which it either achieves significantly superior or competitive performance to the baselines. Further analysis shows that our method is able to generate molecules that lie close to the training distribution yet do not violate the chemical valency rule, demonstrating the effectiveness of the system of SDEs in modeling the node-edge relationships.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Solving the System Of Reverse-time Sdesmentioning

confidence: 99%

Section: *mentioning

confidence: 99%

“…F and e δt L * S with the Trotter theorem (Trotter, 1959), we can approximate the intractable solution e t( L * F + L * S ) as follows (Dockhorn et al, 2021):…”

Section: *mentioning

confidence: 99%

See 2 more Smart Citations

Score-based Generative Modeling of Graphs via the System of Stochastic Differential Equations

Jo¹,

Lee²,

Hwang³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The research question we are interested in in this paper is how to establish a systematic method to stably generate good data from DDPMs in a relatively small number of refinement steps, whether the task is conditional or unconditional. This is a common issue in DDPM studies, and there have been some studies aiming at improving the framework so that efficient sampling is possible (Song et al, 2020a;Dockhorn et al, 2021). In addition, there are also some studies aimed at improving the efficiency of the DDPM sampling, including (Jolicoeur-Martineau et al, 2021;Kong & Ping, 2021;San-Roman et al, 2021;Watson et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

Itô-Taylor Sampling Scheme for Denoising Diffusion Probabilistic Models using Ideal Derivatives

Tachibana¹,

Go²,

Muneyoshi³

et al. 2021

Preprint

View full text Add to dashboard Cite

Denoising Diffusion Probabilistic Models (DDPMs) have been attracting attention recently as a new challenger to popular deep neural generative models including GAN, VAE, etc. However, DDPMs have a disadvantage that they often require a huge number of refinement steps during the synthesis. To address this problem, this paper proposes a new DDPM sampler based on a second-order numerical scheme for stochastic differential equations (SDEs), while the conventional sampler is based on a first-order numerical scheme. In general, it is not easy to compute the derivatives that are required in higher-order numerical schemes. However, in the case of DDPM, this difficulty is alleviated by the trick which the authors call "ideal derivative substitution". The newly derived higher-order sampler was applied to both image and speech generation tasks, and it is experimentally observed that the proposed sampler could synthesize plausible images and audio signals in relatively smaller number of refinement steps.

show abstract

Multi-marginal Schrödinger Bridges

Chen

Conforti

Georgiou

et al. 2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

We consider the problem to identify the most likely flow in phase space, of (inertial) particles under stochastic forcing, that is in agreement with spatial (marginal) distributions that are specified at a set of points in time. The question raised generalizes the classical Schrödinger Bridge Problem (SBP) which seeks to interpolate two specified end-point marginal distributions of overdamped particles driven by stochastic excitation. While we restrict our analysis to second-order dynamics for the particles, the data represents partial (i.e., only positional) information on the flow at multiple time-points. The solution sought, as in SBP, represents a probability law on the space of paths this closest to a uniform prior while consistent with the given marginals. We approach this problem as an optimal control problem to minimize an action integral a la Benamou-Brenier, and derive a time-symmetric formulation that includes a Fisher information term on the velocity field. We underscore the relation of our problem to recent measure-valued splines in Wasserstein space, which is akin to that between SBP and Optimal Mass Transport (OMT). The connection between the two provides a Sinkhorn-like approach to computing measure-valued splines. We envision that interpolation between measures as sought herein will have a wide range of applications in signal/images processing as well as in data science in cases where data have a temporal dimension.

show abstract

Score-Based Generative Modeling with Critically-Damped Langevin Diffusion

Cited by 16 publications

References 28 publications

Score-based Generative Modeling of Graphs via the System of Stochastic Differential Equations

Score-based Generative Modeling of Graphs via the System of Stochastic Differential Equations

Itô-Taylor Sampling Scheme for Denoising Diffusion Probabilistic Models using Ideal Derivatives

Multi-marginal Schrödinger Bridges

Contact Info

Product

Resources

About