ManiTrans: Entity-Level Text-Guided Image Manipulation via Token-wise Semantic Alignment and Generation

Abstract

Existing text-guided image manipulation methods aim to modify the appearance of the image or to edit a few objects in a virtual or simple scenario, which is far from practical applications. In this work, we study a novel task on text-guided image manipulation on the entity level in the real world (eL-TGIM). The task imposes three basic requirements, (1) to edit the entity consistent with the text descriptions, (2) to preserve the entity-irrelevant regions, and (3) to merge the manipulated entity into the image naturally. To this end, we propose an elegant framework, dubbed as SeMani, forming the Semantic Manipulation of real-world images that can not only edit the appearance of entities but also generate new entities corresponding to the text guidance. To solve eL-TGIM, SeMani decomposes the task into two phases: the semantic alignment phase and the image manipulation phase. In the semantic alignment phase, SeMani incorporates a semantic alignment module to locate the entity-relevant region to be manipulated. In the image manipulation phase, SeMani adopts a generative model to synthesize new images conditioned on the entity-irrelevant regions and target text descriptions. We discuss and propose two popular generation processes that can be utilized in SeMani, the discrete auto-regressive generation with transformers and the continuous denoising generation with diffusion models, yielding SeMani-Trans and SeMani-Diff, respectively. We conduct extensive experiments on the real datasets CUB, Oxford, and COCO datasets to verify that SeMani can distinguish the entity-relevant and -irrelevant regions and achieve more precise and flexible manipulation in a zero-shot manner compared with baseline methods. Our codes and models will be released at https://github.com/Yikai-Wang/SeMani.

DE-net: Dynamic Text-Guided Image Editing Adversarial Networks

“…Following previous works (Wang et al 2022;, we employ the Inception Score (IS) (Salimans et al 2016), CLIP-sim, L2-error, and Manipulation Precision (MP) to evaluate the performance. Higher IS means higher fidelity of the edited images.…”

“…But there is no ground truth (humanedited images) in this task, and calculating with source images makes the FID prefer lazy models which do not edit input images. So we adopt the IS to evaluate the image fidelity as previous works (Wang et al 2022;).…”

“…Different from previous models (Dong et al 2017;Nam, Kim, and Kim 2018;Li et al 2020a,b;Wang et al 2022), our DE-Net is based on a novel dynamic text-guided image editing framework. First, it can compose suitable editing processes based on input images and texts dynamically.…”

“…We compare the proposed method with several state-of-theart text-guided image editing models, including TAGAN (Nam, Kim, and Kim 2018), ManiGAN ), ManiTrans (Wang et al 2022). We also modify the state-ofthe-art text-to-image model DF-GAN (Tao et al 2022) and…”

Region-Aware Diffusion for Zero-shot Text-driven Image Editing

“…and image manipulation [1,2,36,39,47]. Computational approaches [5,9,30] for modifying the style and appearance of objects in natural photographs have made remarkable progress, allowing beginner users to accomplish a wide range of editing effects.…”