Self-Distilled StyleGAN: Towards Generation from Internet Photos

Mokady, Ron; Tov, Omer; Yarom, Michal; Lang, Oran; Mosseri, Inbar; Dekel, Tali; Cohen–Or, Daniel; Irani, Michal

doi:10.1145/3528233.3530708

Cited by 26 publications

(11 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Bau et al [7] further demonstrated how to use masks provided by the user, to localize the text-based editing and restrict the change to a specific spatial region. However, while GAN-based image editing approaches succeed on highly-curated datasets [27], e.g., human faces, they struggle over large and diverse datasets.…”

Section: Fixed Attention Maps and Random Seedmentioning

confidence: 99%

Prompt-to-Prompt Image Editing with Cross Attention Control

Hertz¹,

Mokady²,

Tenenbaum³

et al. 2022

Preprint

Self Cite

126

View full text Add to dashboard Cite

Recent large-scale text-driven synthesis models have attracted much attention thanks to their remarkable capabilities of generating highly diverse images that follow given text prompts. Such text-based synthesis methods are particularly appealing to humans who are used to verbally describe their intent. Therefore, it is only natural to extend the text-driven image synthesis to text-driven image editing. Editing is challenging for these generative models, since an innate property of an editing technique is to preserve most of the original image, while in the text-based models, even a small modification of the text prompt often leads to a completely different outcome. State-of-the-art methods mitigate this by requiring the users to provide a spatial mask to localize the edit, hence, ignoring the original structure and content within the masked region. In this paper, we pursue an intuitive prompt-toprompt editing framework, where the edits are controlled by text only. To this end, we analyze a text-conditioned model in depth and observe that the cross-attention layers are the key to controlling the relation between the spatial layout of the image to each word in the prompt. With this observation, we present several applications which monitor the image synthesis by editing the textual prompt only. This includes localized editing by replacing a word, global editing by adding a specification, and even delicately controlling the extent to which a word is reflected in the image. We present our results over diverse images and prompts, demonstrating high-quality synthesis and fidelity to the edited prompts.* Performed this work while working at Google.

show abstract

Section: Fixed Attention Maps and Random Seedmentioning

confidence: 99%

Prompt-to-Prompt Image Editing with Cross Attention Control

Hertz¹,

Mokady²,

Tenenbaum³

et al. 2022

Preprint

Self Cite

126

View full text Add to dashboard Cite

show abstract

“…StyleGAN-XL [147], for example, scales StyleGAN to large datasets with multiple classes of objects. Self-Distilled StyleGAN [148] aims to use internet photo collections for training. These efforts can enable 3D-aware image synthesis for real-world scenarios, but the attendant computational complexity will need to be addressed.…”

Section: Discussionmentioning

confidence: 99%

A Survey on 3D-aware Image Synthesis

Xia¹,

Xue²

2022

Preprint

View full text Add to dashboard Cite

Recent years have seen remarkable progress in deep learning powered visual content creation. This includes 3D-aware generative image synthesis, which produces high-fidelity images in a 3D-consistent manner while simultaneously capturing compact surfaces of objects from pure image collections without the need for any 3D supervision, thus bridging the gap between 2D imagery and 3D reality. The 3D-aware generative models have shown that the introduction of 3D information can lead to more controllable image generation. The task of 3D-aware image synthesis has taken the field of computer vision by storm, with hundreds of papers accepted to top-tier journals and conferences in recent year (mainly the past two years), but there lacks a comprehensive survey of this remarkable and swift progress. Our survey aims to introduce new researchers to this topic, provide a useful reference for related works, and stimulate future research directions through our discussion section. Apart from the presented papers, we aim to constantly update the latest relevant papers along with corresponding implementations at https://weihaox.github.io/awesome-3D-aware-synthesis.

show abstract

“…Generative model zoo. We evaluate on a collection of 133 generative models trained using different techniques including GANs [34,54,56,58,63,74,79,92,92,107,122], diffusion models [24,44,116], MLP-based generative model CIPS [3], and the autoregressive model VQGAN [30]. For evauation we also manually assign ground truth label to each model based on the type of generated images, with 23 labels in total.…”

Section: Methodsmentioning

confidence: 99%

“…Generative models are opensourced at an unprecedented rate of hundreds per month. They use different learning objectives [25,36,44,48,60,85,114,117], training techniques [54,55,63,79,100,106], and network architectures [13,30,57,97]. They are also trained on different datasets [20,79,108,129] for different applications [2,39,65,90,96,103,133,139].…”

Section: Related Workmentioning

confidence: 99%

“…During model retrieval, extracting query features is con- The query "human wearing a pair of glasses" successfully retrieves a GANSketch [122] model finetuned for human faces with glasses. Similarly, for query "a bird that talks", we find a Self-Distilled GAN [79] trained on Internet parrots images. We compare our Gaussian Density retrieval method with two model-description-based baseline methods, Description (text) and Description (CLIP).…”

Section: Originalmentioning

confidence: 99%

See 1 more Smart Citation