TediGAN: Text-Guided Diverse Face Image Generation and Manipulation

Xia, Weihao; Yang, Yujiu; Xue, Jing-Hao; Wu, Baoyuan

doi:10.1109/cvpr46437.2021.00229

Cited by 258 publications

(227 citation statements)

References 24 publications

Supporting

Mentioning

227

Contrasting

Order By: Relevance

“…1 with a Lanvegin-like procedure where we take a gradient step with respect to the classifier probability and then correct this gradient step with the diffusion model. Unlike many GAN-based methods [12,69,93,43,94], D2C does not need to optimize an inversion procedure at evaluation time, and thus the latent value is much faster to compute; D2C is also better at retaining fine-grained features of the original image due to the reconstruction loss.…”

Section: Conditions From Labeled Examplesmentioning

confidence: 99%

“…To perform conditional generation over an unconditional LVGM, most methods assume access to a discriminative model (e.g., a classifier); the latent space of the generator is then modified to change the outputs of the discriminative model. The disciminative model can operate on either the image space [63,67,25] or the latent space [77,94]. For image space discriminative models, plug-and-play generative networks [63] control the attributes of generated images via Langevin dynamics [75]; these ideas are also explored in diffusion models [83].…”

Section: Conditional Generation With Unconditional Modelsmentioning

confidence: 99%

“…For image space discriminative models, plug-and-play generative networks [63] control the attributes of generated images via Langevin dynamics [75]; these ideas are also explored in diffusion models [83]. Image manipulation methods are based on GANs often operate with latent space discriminators [77,94]. However, these methods have some trouble manipulating real images because of imperfect reconstruction [103,7].…”

Section: Conditional Generation With Unconditional Modelsmentioning

confidence: 99%

See 2 more Smart Citations

D2C: Diffusion-Denoising Models for Few-shot Conditional Generation

Sinha¹,

Song²,

Meng³

et al. 2021

Preprint

View full text Add to dashboard Cite

Conditional generative models of high-dimensional images have many applications, but supervision signals from conditions to images can be expensive to acquire. This paper describes Diffusion-Decoding models with Contrastive representations (D2C), a paradigm for training unconditional variational autoencoders (VAEs) for few-shot conditional image generation. D2C uses a learned diffusion-based prior over the latent representations to improve generation and contrastive selfsupervised learning to improve representation quality. D2C can adapt to novel generation tasks conditioned on labels or manipulation constraints, by learning from as few as 100 labeled examples. On conditional generation from new labels, D2C achieves superior performance over state-of-the-art VAEs and diffusion models. On conditional image manipulation, D2C generations are two orders of magnitude faster to produce over StyleGAN2 ones and are preferred by 50% − 60% of the human evaluators in a double-blind study. * Equal contribution.Preprint. Under review.

show abstract

Section: Conditions From Labeled Examplesmentioning

confidence: 99%

Section: Conditional Generation With Unconditional Modelsmentioning

confidence: 99%

Section: Conditional Generation With Unconditional Modelsmentioning

confidence: 99%

See 1 more Smart Citation

D2C: Diffusion-Denoising Models for Few-shot Conditional Generation

Sinha¹,

Song²,

Meng³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Shen et al [51] perform eigenvalue decomposition on the affine transformation layers of StyleGAN2 generators [28] to learn versatile manipulation directions. Xia et al [59] and Patashnik and Wu et al [44] manipulate images using a human-understandable text prompt providing a more intuitive image editing interface.…”

Section: Gan-based Image Editingmentioning

confidence: 99%

StyleFusion: A Generative Model for Disentangling Spatial Segments

Kafri¹,

Patashnik²,

Alaluf³

et al. 2021

Preprint

View full text Add to dashboard Cite

We present StyleFusion, a new mapping architecture for StyleGAN, which takes as input a number of latent codes and fuses them into a single style code. Inserting the resulting style code into a pre-trained StyleGAN generator results in a single harmonized image in which each semantic region is controlled by one of the input latent codes. Effectively, StyleFusion yields a disentangled representation of the image, providing fine-grained control over each region of the generated image. Moreover, to help facilitate global control over the generated image, a special input latent code is incorporated into the fused representation. StyleFusion operates in a hierarchical manner, where each level is tasked with learning to disentangle a pair of image regions (e.g., the car body and wheels). The resulting learned disentanglement allows one to modify both local, fine-grained semantics (e.g., facial features) as well as more global features (e.g., pose and background), providing improved flexibility in the synthesis process. As a natural extension, Style-Fusion enables one to perform semantically-aware crossimage mixing of regions that are not necessarily aligned. Finally, we demonstrate how StyleFusion can be paired with existing editing techniques to more faithfully constrain the edit to the user's region of interest. Code is available at: https://github.com/OmerKafri/StyleFusion.

show abstract

“…With rule-based instructions and predefined semantic labels, they [17,18] first carry out LBIE but under limited practicality. Inspired by textto-image generation [19,20,21], previous works [22,23,24,25,26,27,30] perform LBIE by conditional GAN. Following multi-turn manipulation from humans, iterative LBIE (ILBIE) [28,29] edits images step-by-step.…”

Section: Language-based Image Editingmentioning

confidence: 99%

Language-Driven Image Style Transfer

Wang

2021

Preprint

View full text Add to dashboard Cite

Despite having promising results, style transfer, which requires preparing style images in advance, may result in lack of creativity and accessibility. Following human instruction, on the other hand, is the most natural way to perform artistic style transfer that can significantly improve controllability for visual effect applications. We introduce a new task-language-driven image style transfer (LDIST)-to manipulate the style of a content image, guided by a text. We propose contrastive language visual artist (CLVA) that learns to extract visual semantics from style instructions and accomplish LDIST by the patch-wise style discriminator. The discriminator considers the correlation between language and patches of style images or transferred results to jointly embed style instructions. CLVA further compares contrastive pairs of content image and style instruction to improve the mutual relativeness between transfer results. The transferred results from the same content image can preserve consistent content structures. Besides, they should present analogous style patterns from style instructions that contain similar visual semantics. The experiments show that our CLVA is effective and achieves superb transferred results on LDIST.Project website: https://ai-sub.github.io/ldist/ Preprint. Under review.

show abstract

TediGAN: Text-Guided Diverse Face Image Generation and Manipulation

Cited by 258 publications

References 24 publications

D2C: Diffusion-Denoising Models for Few-shot Conditional Generation

D2C: Diffusion-Denoising Models for Few-shot Conditional Generation

StyleFusion: A Generative Model for Disentangling Spatial Segments

Language-Driven Image Style Transfer

Contact Info

Product

Resources

About