Prompt-to-Prompt Image Editing with Cross Attention Control

Abstract

Recent large-scale text-driven synthesis models have attracted much attention thanks to their remarkable capabilities of generating highly diverse images that follow given text prompts. Such text-based synthesis methods are particularly appealing to humans who are used to verbally describe their intent. Therefore, it is only natural to extend the text-driven image synthesis to text-driven image editing. Editing is challenging for these generative models, since an innate property of an editing technique is to preserve most of the original image, while in the text-based models, even a small modification of the text prompt often leads to a completely different outcome. State-of-the-art methods mitigate this by requiring the users to provide a spatial mask to localize the edit, hence, ignoring the original structure and content within the masked region. In this paper, we pursue an intuitive prompt-toprompt editing framework, where the edits are controlled by text only. To this end, we analyze a text-conditioned model in depth and observe that the cross-attention layers are the key to controlling the relation between the spatial layout of the image to each word in the prompt. With this observation, we present several applications which monitor the image synthesis by editing the textual prompt only. This includes localized editing by replacing a word, global editing by adding a specification, and even delicately controlling the extent to which a word is reflected in the image. We present our results over diverse images and prompts, demonstrating high-quality synthesis and fidelity to the edited prompts.* Performed this work while working at Google.

Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models

“…Another related line of work aims to introduce specific concepts to a pre-trained text-to-image model by learning to map a set of images to a "word" in the embedding space of the model [18,25,41]. Several works have also explored providing users with more control over the synthesis process solely through the use of the input text prompt [8,20,24,46].…”

“…We operate over the 16 × 16 attention units since they have been shown to contain the most semantic information [20].…”

Leveraging Off-the-shelf Diffusion Model for Multi-attribute Fashion Image Manipulation

“…Recently, a line of works [33,28] focus on improving the sampling speed of diffusion model, by either altering the Markovian noising process or embedding the diffusion steps into a learned latent space. Another group [15,3,10] studies the applications of diffusion models such as text-guided image manipulation.…”

“…Blended-diffusion [3] uses a user-provided mask and a textual prompt during the diffusion process to blend the target and the existing background iteratively. A concurrent work of ours, prompt-toprompt [10] captures the text cross-attention structure to enable purely prompt-based scene editing without any explicit masks. In our work, we take blended-diffusion as the starting point, and incorporate a domain-specific classifier and its attention structure for mask-free multi-attribute fashion image manipulation.…”

“…Denoising diffusion models [11,33,7,26,30] have recently gained great attention from the research community for their impressive synthesis quality, training stability and scalability. They have demonstrated promising performances across diverse tasks and benchmarks spanning unconditional image synthesis [7], text-driven image generation [26,30,21], image manipulation [3,10] and video synthesis [13]. Nevertheless, studies on diffusion models are far from complete; unlike traditional generative model * Work done during internship at NAVER.…”