Region-Aware Diffusion for Zero-shot Text-driven Image Editing

Abstract

Fig. 1. The results of the proposed region-aware diffusion model (RDM). The texts adhere to the phrase rule "A → B", indicating that RDM transforms entity A into entity B.Image manipulation under the guidance of textual descriptions has recently received a broad range of attention. In this study, we focus on the regional editing of images with the guidance of given text prompts. Different from current mask-based image editing methods, we propose a novel region-aware diffusion model (RDM) for entity-level image editing, which could automatically locate the region of interest and replace it following given text prompts. To strike a balance between image fidelity and inference speed, we design the intensive diffusion pipeline by combing latent space diffusion and enhanced directional guidance. In addition, to preserve

Multi-Region Text-Driven Manipulation of Diffusion Imagery

“…Recent entity-level editing methods (Huang et al 2023;Hertz et al 2022) have been inspired by exerting control over the latent space or attention maps (Chen, Laina, and Vedaldi 2023). Their constraints on the initial image layout hinder the ability to make substantial structural modifications, not to mention the process of object addition or removal.…”

“…1. To mitigate the interference issue, we employ a pre-trained CLIP segmentation model from RDM (Huang et al 2023), denoted as guidance model Φ, to impose spatial-aware guidance.…”

“…The further editing of real-world images holds value in manipulation. When handling multi-object real images, prevailing inversion methods (Mokady et al 2023;Huberman-Spiegelglas, Kulikov, and Michaeli 2023;Gal et al 2022) guided by textual prompts can only reconstruct and edit in the global region. However, the presence of multiple objectives in the prompts can disrupt the initial layout, causing impractical distortions.…”

M3VSNET: Unsupervised Multi-Metric Multi-View Stereo Network

“…FISEdit [221], Blended Latent Diffusion [222], PFB-Diff [223], DiffEdit [224], RDM [225], MFL [226], Differential Diffusion [227], Watch Your Steps [228], Blended Diffusion [229], ZONE [230], Inpaint Anything [231] Multi-Noise Redirection The Stable Artist [232], SEGA [233], LEDITS [234], OIR-Diffusion [235] Fig. 7: Taxonomy of training and finetuning free approaches for image editing.…”

“…This selective editing approach safeguards unedited regions and preserves their semantic integrity. RDM [225] introduces a region-aware diffusion model that seamlessly integrates masks to automatically pinpoint and edit regions of interest based on text-driven guidance. MFL [226] proposes a two-stage mask-free training paradigm tailored for textguided image editing.…”