Image-Based CLIP-Guided Essence Transfer

Chefer, Hila; Benaim, Sagie; Paiss, Roni; Wolf, Lior

doi:10.48550/arxiv.2110.12427

Cited by 5 publications

(6 citation statements)

References 17 publications

(35 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hence, they fail to achieve the same quality. Chefer et al [2021] utilize CLIP to blend two facial images, demonstrating better preservation of the original identity while successfully transferring meaningful semantic features from the desired target images. find meaningful directions in CLIP-space in an unsupervised manner, map them to latent-space directions, and use CLIP to automatically generate natural language descriptions for these directions.…”

Section: Latent Space Editingmentioning

confidence: 99%

State-of-the-Art in the Architecture, Methods and Applications of StyleGAN

Bermano¹,

Gal²,

Alaluf³

et al. 2022

Preprint

View full text Add to dashboard Cite

Generative Adversarial Networks (GANs) have established themselves as a prevalent approach to image synthesis. Of these, StyleGAN offers a fascinating case study, owing to its remarkable visual quality and an ability to support a large array of downstream tasks. This state-of-the-art report covers the StyleGAN architecture, and the ways it has been employed since its conception, while also analyzing its severe limitations. It aims to be of use for both newcomers, who wish to get a grasp of the field, and for more experienced readers that might benefit from seeing current research trends and existing tools laid out. Among StyleGAN's most interesting aspects is its learned latent space. Despite being learned with no supervision, it is surprisingly well-behaved and remarkably disentangled. Combined with StyleGAN's visual quality, these properties gave rise to unparalleled editing capabilities. However, the control offered by StyleGAN is inherently limited to the generator's learned distribution, and can only be applied to images generated by StyleGAN itself. Seeking to bring StyleGAN's latent control to real-world scenarios, the study of GAN inversion and latent space embedding has quickly gained in popularity. Meanwhile, this same study has helped shed light on the inner workings and limitations of StyleGAN. We map out StyleGAN's impressive story through these investigations, and discuss the details that have made StyleGAN the go-to generator. We further elaborate on the visual priors StyleGAN constructs, and discuss their use in downstream discriminative tasks. Looking forward, we point out StyleGAN's limitations and speculate on current trends and promising directions for future research, such as task and target specific fine-tuning.

show abstract

Section: Latent Space Editingmentioning

confidence: 99%

State-of-the-Art in the Architecture, Methods and Applications of StyleGAN

Bermano¹,

Gal²,

Alaluf³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…These studies have revealed that the latent space is disentangled in different degrees, and therefore is suitable in various tasks. Due to the disentangle property in W and S spaces, large number of works Chefer et al [2021], , Roich et al…”

Section: Feature Disentanglement In Stylegan Latent Spacementioning

confidence: 99%

“…using text-driven latent manipulation , Gal et al [2021], . Furthermore, reference images/videos have also been considered Chefer et al [2021], , Lewis et al [2021] to pinpoint the generation process.…”

Section: Introductionmentioning

confidence: 99%

Grasping the Arrow of Time from the Singularity: Decoding Micromotion in Low-dimensional Latent Spaces from StyleGAN

Wu¹,

Jiang²,

Wu³

et al. 2022

Preprint

View full text Add to dashboard Cite

The disentanglement of StyleGAN latent space has paved the way for realistic and controllable image editing, but does StyleGAN know anything about temporal motion, as it was only trained on static images? To study the motion features in the latent space of StyleGAN, in this paper, we hypothesize and demonstrate that a series of meaningful, natural, and versatile small, local movements (referred to as "micromotion", such as expression, head movement, and aging effect) can be represented in low-rank spaces extracted from the latent space of a conventionally pre-trained StyleGAN-v2 model for face generation, with the guidance of proper "anchors" in the form of either short text or video clips. Starting from one target face image, with the editing direction decoded from the low-rank space, its micromotion features can be represented as simple as an affine transformation over its latent feature. Perhaps more surprisingly, such micromotion subspace, even learned from just single target face, can be painlessly transferred to other unseen face images, even those from vastly different domains (such as oil painting, cartoon, and sculpture faces). It demonstrates that the local feature geometry corresponding to one type of micromotion is aligned across different face subjects, and hence that StyleGAN-v2 is indeed "secretly" aware of the subject-disentangled feature variations caused by that micromotion. We present various successful examples of applying our low-dimensional micromotion subspace technique to directly and effortlessly manipulate faces, showing high robustness, low computational overhead, and impressive domain transferability. Our codes are available at https: //github.com/wuqiuche/micromotion-StyleGAN.

show abstract

“…VQGAN-CLIP [8,9,45] leverage CLIP for text-guided image generation. Concurrent work uses CLIP to fine-tune a pre-trained StyleGAN [11], and for image stylization [6]. Another concurrent work uses the ShapeNet dataset [5] and CLIP to perform unconditional 3D voxel generation [48].…”

Section: Related Workmentioning

confidence: 99%