Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models

Abstract

Figure 1. Given a pre-trained text-to-image diffusion model (e.g. Stable Diffusion [39]) our method, Attend-and-Excite, guides the generative model to modify the cross-attention values during the image synthesis process to generate images that more faithfully depict the input text prompt. Stable Diffusion alone (top row) struggles to generate multiple objects (e.g. a horse and a dog). However, by incorporating Attend-and-Excite (bottom row) to strengthen the subject tokens (marked in blue), we achieve images that are more semantically faithful with respect to the input text prompts.

Cones: Concept Neurons in Diffusion Models for Customized Generation

“…As shown in Fig. A14, when Cones incorporates the Attend-and-Excite (Feng et al, 2022;Chefer et al, 2023) method to address this issue, it generates better results.…”

Progressive Text-to-Image Diffusion with Soft Latent Direction

“…The remarkable advances in this area are driven by the application of state-of-the-art image-generative models, such as auto-regressive (Ramesh et al 2021;Wang et al 2022) and diffusion models (Ramesh et al 2022;Saharia et al 2022;Rombach et al 2022), as well as the availability of large-scale language-image datasets (Sharma et al 2018;Schuhmann et al 2022). However, existing methods face challenges in synthesizing or editing multiple subjects with specific relational and attributive constraints from textual prompts (Chefer et al 2023). The typical defects that oc-cur in the synthesis results are missing entities, and inaccurate inter-object relations, as shown in ??.…”

“…The typical defects that oc-cur in the synthesis results are missing entities, and inaccurate inter-object relations, as shown in ??. Existing work improves the compositional skills of text-to-image synthesis models by incorporating linguistic structures (Feng et al 2022), and attention controls (Hertz et al 2022;Chefer et al 2023) within the diffusion guidance process. Notably, Structured Diffusion (Feng et al 2022) parse a text to extract numerous noun phrases, Attend-and-Excite (Chefer et al 2023) strength attention activations associated with the most marginalized subject token.…”

Building Chinese interdisciplinary research centers : the case of Tsinghua University

“…text-guided solutions have emerged in the field of image editing and produced impressive results [21,25,40,7,17]. The powerful generative capabilities of diffusion models enable the generation of numerous high-quality images.…”

“…These models can generate high-quality synthetic images based on text prompts, enabling text-guided image editing and producing impressive results. As a result, numerous text-based image editing methods [36,13,10,7,28,8,35] have emerged and evolved. However, such models cannot mimic specific subject characteristics.…”