Key-Locked Rank One Editing for Text-to-Image Personalization

Tewel, Yoad; Gal, Ran; Chechik, Gal; Atzmon, Yuval

doi:10.1145/3588432.3591506

Cited by 30 publications

(3 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This occurs because the method primarily emphasizes the reconstruction loss while disregarding the compositional aspect of the target concept in relation to others. Similar findings are reported in (Tewel et al 2023) which suggests the dominance of the inverted concepts in the generation process encroaches upon the spotlight of other concepts. However, this is simply attributed to an over-fitting problem, with underlying rationale remaining unexplored.…”

Section: Introductionsupporting

confidence: 84%

“…However, the remaining methods, although employing similar approaches of searching for inverted embeddings, rely on either retraining or fine-tuning for this purpose. For instance, DreamBooth (Ruiz et al 2023) retrains the entire Imagen for constructing embeddings for CoI, while Custom Diffusion (Kumari et al 2023), Perfusion (Tewel et al 2023), SVDiff (Han et al 2023), and Cones (Liu et al 2023) only fine-tune partial parameters of the Stable Diffusion model. To mitigate language drift and overfitting problems, a large number of images from the same CoI class are typically utilized as regularization during the training/fine-tuning process.…”

Section: Inversion For Customization and Personalizationmentioning

confidence: 99%

See 1 more Smart Citation

Compositional Inversion for Stable Diffusion Models

Zhang,

Wei,

et al. 2024

AAAI

View full text Add to dashboard Cite

Inversion methods, such as Textual Inversion, generate personalized images by incorporating concepts of interest provided by user images. However, existing methods often suffer from overfitting issues, where the dominant presence of inverted concepts leads to the absence of other desired concepts. It stems from the fact that during inversion, the irrelevant semantics in the user images are also encoded, forcing the inverted concepts to occupy locations far from the core distribution in the embedding space. To address this issue, we propose a method that guides the inversion process towards the core distribution for compositional embeddings. Additionally, we introduce a spatial regularization approach to balance the attention on the concepts being composed. Our method is designed as a post-training approach and can be seamlessly integrated with other inversion methods. Experimental results demonstrate the effectiveness of our proposed approach in mitigating the overfitting problem and generating more diverse and balanced compositions of concepts in the synthesized images. The source code is available at https://github.com/zhangxulu1996/Compositional-Inversion.

show abstract

Section: Introductionsupporting

confidence: 84%

Section: Inversion For Customization and Personalizationmentioning

confidence: 99%

Compositional Inversion for Stable Diffusion Models

Zhang,

Wei,

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…SVDiff [17] and Lightweight DreamBooth [40] further refine this process using Singular Value Decomposition (SVD) and orthogonal incomplete basis within LoRA weight-space respectively. Perfusion [45] incorporates Rank-One Model Editing (ROME) [30], exemplifies targeted model edits aligned with conceptual directions. Meanwhile, HyperDreamBooth [40] enables rapid adaptation to new concepts through hypernetwork-initialized rank-1 residuals.…”

Section: T2i Personalizationmentioning

confidence: 99%

Style transfer network for complex multi-stroke text

Chen

Wang

et al. 2023

Multimedia Systems

View full text Add to dashboard Cite

Neural style transfer has achieved success in many tasks. It is also introduced to text style transfer, which uses a style image to generate transferred images with textures and shapes consistent with the semantic content of the reference image. However, when the text structure is complex, existing methods will encounter problems such as stroke adhesion and unclear text edges. This will affect the aesthetics of the generated image and bring a lot of extra workload to the designers. This paper proposes an improved text style transfer network for complex multistroke texts. We use Shape-matching Generative Adversarial Network (GAN) as the baseline network and perform the following modifications: (1) Morphological methods, erosion and dilation, are introduced in image processing; (2) Residual modules and loss functions are added to the network; and (3) AdaBelief optimizer is adopted to constrain the transfer of text structure. Further, a new dataset of traditional Chinese characters is constructed to train the model. Experimental results show that the proposed method outperforms state-of-the-art methods on both simple characters and complex multi-stroke characters. It is shown that our method increases the readability and aesthetics of the text.

show abstract