Image Super-Resolution via Iterative Refinement

Saharia, Chitwan; Ho, Jonathan; Chan, William; Salimans, Tim; Fleet, David J.; Norouzi, Mohammad

doi:10.48550/arxiv.2104.07636

Cited by 71 publications

(137 citation statements)

References 0 publications

Supporting

Mentioning

130

Contrasting

Order By: Relevance

“…More recently, denoising diffusion models were used to solve inverse problems in both supervised (i.e., degradation model is known during training) (Saharia et al, 2021b;a;Chung et al, 2021;Whang et al, 2021) and unsupervised settings (Kadkhodaie & Simoncelli, 2021;Kawar et al, 2021a;Jalal et al, 2021b;Song et al, 2021b;c;Choi et al, 2021). Unlike previous approaches, diffusion-based methods can successfully recover images from measurements with significant noise.…”

Section: Related Workmentioning

confidence: 99%

Denoising Diffusion Restoration Models

Kawar¹,

Elad²,

Ermon³

et al. 2022

Preprint

View full text Add to dashboard Cite

Many interesting tasks in image restoration can be cast as linear inverse problems. A recent family of approaches for solving these problems uses stochastic algorithms that sample from the posterior distribution of natural images given the measurements. However, efficient solutions often require problem-specific supervised training to model the posterior, whereas unsupervised methods that are not problem-specific typically rely on inefficient iterative methods. This work addresses these issues by introducing Denoising Diffusion Restoration Models (DDRM), an efficient, unsupervised posterior sampling method. Motivated by variational inference, DDRM takes advantage of a pre-trained denoising diffusion generative model for solving any linear inverse problem. We demonstrate DDRM's versatility on several image datasets for super-resolution, deblurring, inpainting, and colorization under various amounts of measurement noise. DDRM outperforms the current leading unsupervised methods on the diverse ImageNet dataset in reconstruction quality, perceptual quality, and runtime, being 5× faster than the nearest competitor. DDRM also generalizes well for natural images out of the distribution of the observed ImageNet training set. Our code is available at this GitHub repository.

show abstract

Section: Related Workmentioning

confidence: 99%

Denoising Diffusion Restoration Models

Kawar¹,

Elad²,

Ermon³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…A similar parameterization has been considered in recent work on conditional DDPM models [22,47]. We concatenate the VAE reconstruction to the reverse process representation x t at each time step t to obtain x t−1 .…”

Section: Formulationmentioning

confidence: 99%

“…Following the seminal work of [21,50] on diffusion models, there has been a lot of recent progress in both unconditional [13,30,38] and conditional diffusion models [8,11,22,47] (including score-based models [52,54], based on a connection proposed in [21]) for a variety of downstream tasks including image synthesis, audio synthesis and likelihood estimation among others. Recent work in DDPMs also includes speeding up the DDPM sampling process [35,51,61].…”

Section: Related Workmentioning

confidence: 99%

“…Conditional DDPM: Conditional DDPM as introduced in [22] and [47] uses multiple diffusion models for generating high-resolution images in a cascaded fashion. However, for even a two-stage pipeline, the sampling time of such models would be effectively much higher than DiffuseVAE.…”

Section: Related Workmentioning

confidence: 99%

“…However, the extension of this form of conditioning for images remains under-explored in diffusion models. [47] explored continuous noise conditioning for images in the context of image super-resolution but do not provide much analysis in terms of quantitative measures.…”

Section: Continuous Noise Conditioning In Diffusevaementioning

confidence: 99%

See 2 more Smart Citations

DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latents

Pandey¹,

Mukherjee²,

Rai³

et al. 2022

Preprint

View full text Add to dashboard Cite

Diffusion Probabilistic models have been shown to generate state-of-the-art results on several competitive image synthesis benchmarks but lack a low-dimensional, interpretable latent space, and are slow at generation. On the other hand, Variational Autoencoders (VAEs) typically have access to a low-dimensional latent space but exhibit poor sample quality. Despite recent advances, VAEs usually require high-dimensional hierarchies of the latent codes to generate high-quality samples. We present DiffuseVAE, a novel generative framework that integrates VAE within a diffusion model framework, and leverage this to design a novel conditional parameterization for diffusion models. We show that the resulting model can improve upon the unconditional diffusion model in terms of sampling efficiency while also equipping diffusion models with the low-dimensional VAE inferred latent code. Furthermore, we show that the proposed model can generate high-resolution samples and exhibits synthesis quality comparable to state-of-the-art models on standard benchmarks. Lastly, we show that the proposed method can be used for controllable image synthesis and also exhibits out-of-the-box capabilities for downstream tasks like image super-resolution and denoising. For reproducibility, our source code is publicly available at https://github.com/kpandey008/DiffuseVAE. 1

show abstract