Diffusion Autoencoders: Toward a Meaningful and Decodable Representation

Preechakul, Konpat; Chatthee, Nattanat; Wizadwongsa, Suttisak; Suwajanakorn, Supasorn

doi:10.48550/arxiv.2111.15640

Cited by 6 publications

(9 citation statements)

References 35 publications

(116 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The computational benefits of using diffusion to model a latent space has been noted by previous works. Preechakul et al [38] propose an autoencoder framework where diffusion models are used to render latent variables as images, and a second diffusion model is used to generate these latents (similar to our diffusion prior). Vahdat et al [51] use a score-based model for the latent space of a VAE, while Rombach et al [42] use diffusion models on the latents obtained from a VQGAN [14] like autoencoder.…”

Section: Related Workmentioning

confidence: 99%

Hierarchical Text-Conditional Image Generation with CLIP Latents

Ramesh¹,

Dhariwal²,

Nichol³

et al. 2022

Preprint

703

845

View full text Add to dashboard Cite

Contrastive models like CLIP have been shown to learn robust representations of images that capture both semantics and style. To leverage these representations for image generation, we propose a two-stage model: a prior that generates a CLIP image embedding given a text caption, and a decoder that generates an image conditioned on the image embedding. We show that explicitly generating image representations improves image diversity with minimal loss in photorealism and caption similarity. Our decoders conditioned on image representations can also produce variations of an image that preserve both its semantics and style, while varying the non-essential details absent from the image representation. Moreover, the joint embedding space of CLIP enables language-guided image manipulations in a zero-shot fashion. We use diffusion models for the decoder and experiment with both autoregressive and diffusion models for the prior, finding that the latter are computationally more efficient and produce higher-quality samples.

show abstract

Section: Related Workmentioning

confidence: 99%

Hierarchical Text-Conditional Image Generation with CLIP Latents

Ramesh¹,

Dhariwal²,

Nichol³

et al. 2022

Preprint

703

845

View full text Add to dashboard Cite

show abstract

“…However, manipulating the latent variables of DMs directly may lead to distorted images or incorrect manipulation [18], as they lack high-level semantic information. Some works [23,37] construct an external semantic latent space to address this issue. Asyrp [20] explores the deepest bottleneck of the UNet as a local semantic latent space (h-space) to accommodate semantic image manipulation.…”

Section: Introductionmentioning

confidence: 99%

Clinical value of the emergency department in screening and diagnosis of COVID-19 in China

Zhang

Pan

Zhao

et al. 2020

J. Zhejiang Univ. Sci. B

View full text Add to dashboard Cite

Studying the implications and characterizations of the excited state quantum phase transitions (ESQPTs) would enable us to understand various phenomena observed in quantum many body systems. In this work, we delve into the affects and characterizations of the ESQPTs in the anharmonic Lipkin-Meshkov-Glick (LMG) model by means of the entropy of the quantum work distribution. The entropy of the work distribution measures the complexity of the work distribution and behaves as a valuable tool for analyzing nonequilibrium work statistics. We show that the entropy of the work distribution captures salient signatures of the underlying ESQPTs in the model. In particular, a detailed analyses of the scaling behavior of the maximal entropy verifies that it acts as a witness of the ESQPTs. We further demonstrate that the entropy of the work distribution also reveals the features of the ESQPTs in the energy space and can be used to determine their critical energies. Our results provide further evidence of the usefulness of the entropy of the work distribution for investigating various phase transitions in quantum many body systems and open up a promising way for experimentally exploring the signatures of ESQPTs.

show abstract

“…Some works find that combining DDPMs with VAEs can help accelerate the sampling process of DDPMs. Preechakul et al (2021) condition the reverse process of the DDPM on an encoded vector of an image. Pandey et al (2022) condition the reverse process of the DDPM on an reconstructed image by a VAE.…”

Section: Related Workmentioning

confidence: 99%

“…In this way, we can control high-level semantics for our combined model. Other works (Preechakul et al, 2021;Pandey et al, 2022) have also shown that it is possible to combine DDPMs and VAEs to control high-level semantics of the generated images. Compared with them, our method can achieve better sample quality using the same number of denoising steps as shown in Table 2 and Table 6.…”

Section: A2 Conditional Generationmentioning

confidence: 99%

“…Although the number of denoising steps is reduced, such integrated denoising steps break the Gaussian assumption of DDPMs' denoising process (Xiao et al, 2021;Sohl-Dickstein et al, 2015), resulting in dropped sample quality. On the other hand, acceleration of DDPMs is also indirectly achieved by (Preechakul et al, 2021;Pandey et al, 2022), which inject image features or VAE-generated images into the denoising process as additional conditions. Such conditioned denoising processes are more effective than the original one, requiring fewer denoising steps to obtain a high-quality sample.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Accelerating Diffusion Models via Early Stop of the Diffusion Process

Zhaoyang¹,

XU²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Denoising Diffusion Probabilistic Models (DDPMs) have achieved impressive performance on various generation tasks. By modeling the reverse process of gradually diffusing the data distribution into a Gaussian distribution, generating a sample in DDPMs can be regarded as iteratively denoising a randomly sampled Gaussian noise. However, in practice DDPMs often need hundreds or even thousands of denoising steps to obtain a high-quality sample from the Gaussian noise, leading to extremely low inference efficiency. In this work, we propose a principled acceleration strategy, referred to as Early-Stopped DDPM (ES-DDPM), for DDPMs. The key idea is to stop the diffusion process early where only the few initial diffusing steps are considered and the reverse denoising process starts from a non-Gaussian distribution. By further adopting a powerful pre-trained generative model, such as GAN and VAE, in ES-DDPM, sampling from the target non-Gaussian distribution can be efficiently achieved by diffusing samples obtained from the pre-trained generative model. In this way, the number of required denoising steps is significantly reduced. In the meantime, the sample quality of ES-DDPM also improves substantially, outperforming both the vanilla DDPM and the adopted pre-trained generative model. On extensive experiments across CIFAR-10, CelebA, ImageNet, LSUN-Bedroom and LSUN-Cat, ES-DDPM obtains promising acceleration effect and performance improvement over representative baseline methods. Moreover, ES-DDPM also demonstrates several attractive properties, including being orthogonal to existing acceleration methods, as well as simultaneously enabling both global semantic and local pixel-level control in image generation.Preprint. Under review.

show abstract

Diffusion Autoencoders: Toward a Meaningful and Decodable Representation

Cited by 6 publications

References 35 publications

Hierarchical Text-Conditional Image Generation with CLIP Latents

Hierarchical Text-Conditional Image Generation with CLIP Latents

Clinical value of the emergency department in screening and diagnosis of COVID-19 in China

Accelerating Diffusion Models via Early Stop of the Diffusion Process

Contact Info

Product

Resources

About