Open-Edit: Open-Domain Image Manipulation with Open-Vocabulary Instructions

Liu, Xihui; Lin, Zhe; Zhang, Jianming; Zhao, Handong; Tran, Quan Hung; Wang, Xiaogang; Li, Hongsheng

doi:10.1007/978-3-030-58621-8_6

Cited by 27 publications

(14 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hence, the data domain that the style-GAN is pretrained on will limit the editing domain. Although [25] trains a generator by reconstruction and thus can work for any open image domain, the generation quality is not guaranteed. In contrast, the editing quality of our method is guaranteed by the unique properties of our learned editing space.…”

Section: Related Workmentioning

confidence: 99%

“…Language is a convenient way to incorporate user's editing intention, which is a more intuitive and convenient interface than existing operation-based editing interfaces. Given our pretrained generator, we solve the LGIE tasks by finding a mapping between the text input and our low-dimensional editing space, which is a different framework compared to previous works [2,4,7,9,18,23,25,28,29,37,38,49]. Next, we describe our approaches for both supervised LGIE as well as zero-shot LGIE.…”

Section: Language-guided Image Editingmentioning

confidence: 99%

“…For zero-shot LGIE, as it works for open-domain image and open-vocabulary requests, we compare the qualitative performance on given examples with two other SOTA methods -OpenEdit [25] and StyleCLIP [31]. OpenEdit has no constraint for both image and request, while StyleCLIP can only work for close-domain images.…”

Section: Language-guided Image Editingmentioning

confidence: 99%

“…Image editing has shown wide spectrum of applications in various scenarios including image retouching [12,39], style transfer [47,48], language-guided image editing [18,23,25,38], image harmonization [11], colorization [50], etc. However, the current research landscape independently studies these tasks on small and diverse datasets, underscoring the commonality of the image editing required for each task.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

SpaceEdit: Learning a Unified Editing Space for Open-Domain Image Editing

Shi¹,

Xu²,

Zheng³

et al. 2021

Preprint

View full text Add to dashboard Cite

Recently, large pretrained models (e.g., BERT, Style-GAN, CLIP) have shown great knowledge transfer and generalization capability on various downstream tasks within their domains. Inspired by these efforts, in this paper we propose a unified model for open-domain image editing focusing on color and tone adjustment of open-domain images while keeping their original content and structure. Our model learns a unified editing space that is more semantic, intuitive, and easy to manipulate than the operation space (e.g., contrast, brightness, color curve) used in many existing photo editing softwares. Our model belongs to the image-to-image translation framework which consists of an image encoder and decoder, and is trained on pairs of before-and after-images to produce multimodal outputs. We show that by inverting image pairs into latent codes of the learned editing space, our model can be leveraged for various downstream editing tasks such as language-guided image editing, personalized editing, editing-style clustering, retrieval, etc. We extensively study the unique properties of the editing space in experiments and demonstrate superior performance on the aforementioned tasks.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Language-guided Image Editingmentioning

confidence: 99%

Section: Language-guided Image Editingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

SpaceEdit: Learning a Unified Editing Space for Open-Domain Image Editing

Shi¹,

Xu²,

Zheng³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…It should preserve the structure of car scene from the content image and simultaneously modifies the style pattern that corresponds to "horizontally lined texture, blues." Our LDIST task is different from the general language-based image-editing (LBIE) [25,26,30,28], which usually alters objects or properties of object in an image. The main challenge of LDIST is to express visual semantics from language as style patterns.…”

Section: Introductionmentioning

confidence: 99%

Language-Driven Image Style Transfer

Wang

2021

Preprint

View full text Add to dashboard Cite

Despite having promising results, style transfer, which requires preparing style images in advance, may result in lack of creativity and accessibility. Following human instruction, on the other hand, is the most natural way to perform artistic style transfer that can significantly improve controllability for visual effect applications. We introduce a new task-language-driven image style transfer (LDIST)-to manipulate the style of a content image, guided by a text. We propose contrastive language visual artist (CLVA) that learns to extract visual semantics from style instructions and accomplish LDIST by the patch-wise style discriminator. The discriminator considers the correlation between language and patches of style images or transferred results to jointly embed style instructions. CLVA further compares contrastive pairs of content image and style instruction to improve the mutual relativeness between transfer results. The transferred results from the same content image can preserve consistent content structures. Besides, they should present analogous style patterns from style instructions that contain similar visual semantics. The experiments show that our CLVA is effective and achieves superb transferred results on LDIST.Project website: https://ai-sub.github.io/ldist/ Preprint. Under review.

show abstract