Denoising Diffusion Restoration Models

Kawar, Bahjat; Elad, Michael; Ermon, Stefano; Song, Jiaming

doi:10.48550/arxiv.2201.11793

Cited by 26 publications

(53 citation statements)

References 32 publications

(48 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Score-based diffusion models have been used for image editing [Meng et al, 2022, Saharia et al, 2021 and our approach to video generation might be viewed as an analogy to classical image inpainting, but in the temporal dimension. The GLIDE or Guided Language to Image Diffusion for Generation and Editing approach of uses CLIP-guided diffusion for image editing, while Denoising Diffusion Restoration Models (DDRM) Kawar et al [2022] additionally condition on a corrupted image to restore the clean image. Adversarial variants of score-based diffusion models have been used to enhance quality [Jolicoeur-Martineau et al, 2021a] or speed [Xiao et al, 2021].…”

Section: Related Workmentioning

confidence: 99%

MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation

Voleti¹,

Jolicoeur-Martineau²,

Pal³

2022

Preprint

View full text Add to dashboard Cite

Video prediction is a challenging task. The quality of video frames from current state-of-the-art (SOTA) generative models tends to be poor and generalization beyond the training data is difficult. Furthermore, existing prediction frameworks are typically not capable of simultaneously handling other video-related tasks such as unconditional generation or interpolation. In this work, we devise a generalpurpose framework called Masked Conditional Video Diffusion (MCVD) for all of these video synthesis tasks using a probabilistic conditional score-based denoising diffusion model, conditioned on past and/or future frames. We train the model in a manner where we randomly and independently mask all the past frames or all the future frames. This novel but straightforward setup allows us to train a single model that is capable of executing a broad range of video tasks, specifically: future/past prediction -when only future/past frames are masked; unconditional generation -when both past and future frames are masked; and interpolation -when neither past nor future frames are masked. Our experiments show that this approach can generate high-quality frames for diverse types of videos. Our MCVD models are built from simple non-recurrent 2D-convolutional architectures, conditioning on blocks of frames and generating blocks of frames. We generate videos of arbitrary lengths autoregressively in a block-wise manner. Our approach yields SOTA results across standard video prediction and interpolation benchmarks, with computation times for training models measured in 1-12 days using ≤ 4 GPUs. Project page: https://mask-cond-video-diffusion.github.io Code: https://github.com/voletiv/mcvd-pytorch * Equal Contribution Preprint. Under review.

show abstract

Section: Related Workmentioning

confidence: 99%

MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation

Voleti¹,

Jolicoeur-Martineau²,

Pal³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Menick and Kalchbrenner [53] suggest a similar method based on resolution and bit-depth upscaling, although in a non-parallel way. Previously, generative models, including diffusion models, have been utilized for image deblurring [45,46,1,80], super-resolution [48,65,14,57,7,64,12], and other types of inverse problems [38,11,32,74,10,39]. While our model does effectively perform deblurring/superresolution, the main difference to these works is that instead of using a pre-existing generative model to solve the inverse problem, we do the exact opposite and create a new generative model through iteratively solving an inverse problem using a loss function that directly reverses the heat equation.…”

Section: Related Workmentioning

confidence: 99%

Generative Modelling With Inverse Heat Dissipation

Rissanen¹,

Heinonen²,

Solin³

2022

Preprint

View full text Add to dashboard Cite

While diffusion models have shown great success in image generation, their noiseinverting generative process does not explicitly consider the structure of images, such as their inherent multi-scale nature. Inspired by diffusion models and the desirability of coarse-to-fine modelling, we propose a new model that generates images through iteratively inverting the heat equation, a PDE that locally erases fine-scale information when run over the 2D plane of the image. In our novel methodology, the solution of the forward heat equation is interpreted as a variational approximation in a directed graphical model. We demonstrate promising image quality and point out emergent qualitative properties not seen in diffusion models, such as disentanglement of overall colour and shape in images and aspects of neural network interpretability. Spectral analysis on natural images positions our model as a type of dual to diffusion models and reveals implicit inductive biases in them.Preprint. Under review.

show abstract

“…In order to match the definition of the generalized diffusion, we propose to factor the symmetric matrix W by eigenvalue-decomposition W = ŨD ŨT and subsequently W i = ŨD f (i) ŨT . We employ the memory-efficicent eigen-decompisition by Kawar et al [2022] (see Appendix D of DDRM). This leads us to the following proposition Proposition 1.…”

Section: Blur Diffusionmentioning

confidence: 99%