More Control for Free! Image Synthesis with Semantic Diffusion Guidance

Abstract

Controllable image synthesis models allow creation of diverse images based on text instructions or guidance from an example image. Recently, denoising diffusion probabilistic models have been shown to generate more realistic imagery than prior methods, and have been successfully demonstrated in unconditional and class-conditional settings. We explore fine-grained, continuous control of this model class, and introduce a novel unified framework for semantic diffusion guidance, which allows either language or image guidance, or both. Guidance is injected into a pretrained unconditional diffusion model using the gradient of image-text or image matching scores. We explore CLIPbased textual guidance as well as both content and stylebased image guidance in a unified form. Our text-guided synthesis approach can be applied to datasets without associated text annotations. We conduct experiments on FFHQ and LSUN datasets, and show results on fine-grained textguided image synthesis, synthesis of images related to a style or content example image, and examples with both textual and image guidance. 1 1 Project page xh-liu.github.io/sdg/

Imagic: Text-Based Real Image Editing with Diffusion Models

“…It was also suggested to edit images by synthesizing data in user-provided masks, while keeping the rest of the image intact [6,33]. Liu et al [31] guide a diffusion process with a text and an image, synthesising images similar to the given one, and aligned with the given text. Hertz et al [16] alter a text-to-image diffusion process by manipulating cross-attention layers, providing more fine-grained control over generated images, and can edit real images in cases where DDIM inversion provides meaningful attention maps.…”

Blended Latent Diffusion

“…Another line of works is based on CLIP ], a vision-language model that learns a rich shared embedding space for images and text, by contrastive training on a dataset of 400 million (image, text) pairs collected from the internet. Some of them Crowson et al 2022;Crowson 2021;Liu et al 2021;Kim and Ye 2021;Murdock 2021;Paiss et al 2022] combine a pretrained generative model [Brock et al 2018;Esser et al 2021a;] with a CLIP model to steer the generative model to perform text-to-image synthesis. Utilizing CLIP along with a generative model was also used for text-based domain adaptation [Gal et al 2021] and text-to-image without training on text data [Zhou et al 2021;Wang et al 2022;].…”

UPainting: Unified Text-to-Image Diffusion Generation with Cross-modal Guidance

“…Image-text matching models, typically CLIPs, have been extensively utilized to steer image generation models towards text conditions [Liu et al, 2021]. Based on CLIP, Ramesh et al [2022] propose a two-stage diffusion model DALL-E 2, where a prior generates a CLIP image embedding given a text caption, and a decoder generates an image conditioned on the image embedding.…”