MyStyle: A Personalized Generative Prior

Abstract

Figure 1. Using our personalized prior tuned with images of Michelle Obama, we solve various challenging tasks while faithfully preserving her key facial characteristic. Left to right: inpainting, super-resolution, and semantic editing (smile). Each example shows the original input image of Obama, which may be corrupted (top left), and the output based on our personalized face prior (right), compared to a generic face prior (bottom left). The generic face prior is learned from a diverse set of images and produces results that do not preserve Obama's key facial characteristics.

An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion

“…More recently, personalization efforts can also be found in vision and graphics. There it is typical to apply a delicate tuning of a generative model to better reconstruct specific faces or scenes (Bau et al, 2019;Roich et al, 2021;Dinh et al, 2022;Cao et al, 2022;Nitzan et al, 2022).…”

Domain Expansion of Image Generators

“…A prominent line of works have sought to make the target domain inherit knowledge from the source domain [1,2,5,14,17,[21][22][23]31,33,42,43,51]. This approach allows generalization beyond the target domain per-se and is especially useful when training data is scarce.…”

“…The domain adaptation objective is applied to images generated from latent codes z ∈ Z, sampled from distribution p(z) defined on the entire space Z. Commonly the distribution is a Gaussian, or is derived from it [12] but some exceptions exist [21,43]. Our strategy is to transform this sample distribution into one restricted to the affine subspace Z i .…”

“…We adopt StyleGAN2 [13] as the source generator architecture, for its disentangled latent space and because it has been the dominant test bed for generative domain adaptation methods in recent years [1,5,21,22,43,51].…”

“…Recent domain adaptation techniques piggyback on the tremendous success of modern generative image models [3,12,32,40], by adapting a pretrained generator so it can generate images from a new target domain. Oftentimes, the target domain is defined with respect to the source domain [5,21,22], e.g., changing the "stylization" from a photorealistic image to a sketch. When such a relationship holds, domain adaptation typically seeks to preserve the factors of variations learned in the source domain, and transfer them to the new one (e.g., making the human depicted in a sketch smile based on the prior from a face generator).…”