Anycost GANs for Interactive Image Synthesis and Editing

Lin, Ji; Zhang, Richard; Ganz, Frieder; Han, Song; Zhu, Jun‐Yan

doi:10.1109/cvpr46437.2021.01474

Cited by 65 publications

(34 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In supplementary material, we investigate classification accuracy as a function of the number of optimization steps, and also perform experiments using an alternative inversion method [75] on a smaller face GAN, obtaining similar results. Moreover, recent alternative architectures trained specifically for efficiency [36] or invertibility [44] may help further reduce the computational cost of image reconstruction.…”

Section: Discussionmentioning

confidence: 99%

Ensembling with Deep Generative Views

Chai¹,

Zhu²,

Shechtman³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Recent generative models can synthesize "views" of artificial images that mimic real-world variations, such as changes in color or pose, simply by learning from unlabeled image collections. Here, we investigate whether such views can be applied to real images to benefit downstream analysis tasks such as image classification. Using a pretrained generator, we first find the latent code corresponding to a given real input image. Applying perturbations to the code creates natural variations of the image, which can then be ensembled together at test-time. We use StyleGAN2 as the source of generative augmentations and investigate this setup on classification tasks involving facial attributes, cat faces, and cars. Critically, we find that several design decisions are required towards making this process work; the perturbation procedure, weighting between the augmentations and original image, and training the classifier on synthesized images can all impact the result. Currently, we find that while test-time ensembling with GAN-based augmentations can offer some small improvements, the remaining bottlenecks are the efficiency and accuracy of the GAN reconstructions, coupled with classifier sensitivities to artifacts in GAN-generated images.

show abstract

Section: Discussionmentioning

confidence: 99%

Ensembling with Deep Generative Views

Chai¹,

Zhu²,

Shechtman³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…For editing images in W, we use the official implementation of InterFaceGAN [60] for training the linear boundaries using off-the-shelf classifiers. We apply HopeNet [59] for pose, Rothe et al [57,58] for age, and the classifier from Lin et al [41] for the remaining attributes.…”

Section: Editing: Additional Detailsmentioning

confidence: 99%

Third Time's the Charm? Image and Video Editing with StyleGAN3

Alaluf¹,

Patashnik²,

Wu³

et al. 2022

Preprint

View full text Add to dashboard Cite

StyleGAN is arguably one of the most intriguing and well-studied generative models, demonstrating impressive performance in image generation, inversion, and manipulation. In this work, we explore the recent StyleGAN3 architecture, compare it to its predecessor, and investigate its unique advantages, as well as drawbacks. In particular, we demonstrate that while StyleGAN3 can be trained on unaligned data, one can still use aligned data for training, without hindering the ability to generate unaligned imagery. Next, our analysis of the disentanglement of the different latent spaces of StyleGAN3 indicates that the commonly used W/W+ spaces are more entangled than their StyleGAN2 counterparts, underscoring the benefits of using the StyleSpace for fine-grained editing. Considering image inversion, we observe that existing encoder-based techniques struggle when trained on unaligned data. We therefore propose an encoding scheme trained solely on aligned data, yet can still invert unaligned images. Finally, we introduce a novel video inversion and editing workflow that leverages the capabilities of a fine-tuned Style-GAN3 generator to reduce texture sticking and expand the field of view of the edited video. Code is available on our project page: https://yuval-alaluf.github.io/ stylegan3-editing/.* Denotes equal contribution.

show abstract

“…Parallel to these improvements, various works sought to identify areas in which StyleGAN could be improved. Lin et al [2021] note that the high computational cost of full-resolution image generation makes it impractical to utilize the network for interactive editing on edge devices. They proposed an elastic generator architecture that could produce previews at lower resolutions while retaining the same latent semantics.…”

Section: Stylegan Architecturesmentioning

confidence: 99%

State-of-the-Art in the Architecture, Methods and Applications of StyleGAN

Bermano¹,

Gal²,

Alaluf³

et al. 2022

Preprint

View full text Add to dashboard Cite

Generative Adversarial Networks (GANs) have established themselves as a prevalent approach to image synthesis. Of these, StyleGAN offers a fascinating case study, owing to its remarkable visual quality and an ability to support a large array of downstream tasks. This state-of-the-art report covers the StyleGAN architecture, and the ways it has been employed since its conception, while also analyzing its severe limitations. It aims to be of use for both newcomers, who wish to get a grasp of the field, and for more experienced readers that might benefit from seeing current research trends and existing tools laid out. Among StyleGAN's most interesting aspects is its learned latent space. Despite being learned with no supervision, it is surprisingly well-behaved and remarkably disentangled. Combined with StyleGAN's visual quality, these properties gave rise to unparalleled editing capabilities. However, the control offered by StyleGAN is inherently limited to the generator's learned distribution, and can only be applied to images generated by StyleGAN itself. Seeking to bring StyleGAN's latent control to real-world scenarios, the study of GAN inversion and latent space embedding has quickly gained in popularity. Meanwhile, this same study has helped shed light on the inner workings and limitations of StyleGAN. We map out StyleGAN's impressive story through these investigations, and discuss the details that have made StyleGAN the go-to generator. We further elaborate on the visual priors StyleGAN constructs, and discuss their use in downstream discriminative tasks. Looking forward, we point out StyleGAN's limitations and speculate on current trends and promising directions for future research, such as task and target specific fine-tuning.

show abstract

Anycost GANs for Interactive Image Synthesis and Editing

Cited by 65 publications

References 41 publications

Ensembling with Deep Generative Views

Ensembling with Deep Generative Views

Third Time's the Charm? Image and Video Editing with StyleGAN3

State-of-the-Art in the Architecture, Methods and Applications of StyleGAN

Contact Info

Product

Resources

About