Pivotal Tuning for Latent-based Editing of Real Images

Abstract

Recently, a surge of advanced facial editing techniques have been proposed that leverage the generative power of a pre-trained StyleGAN. To successfully edit an image this way, one must first project (or invert) the image into the pre-trained generator's domain. As it turns out, however, StyleGAN's latent space induces an inherent tradeoff between distortion and editability, i.e. between maintaining the original appearance and convincingly altering some of its attributes. Practically, this means it is still challenging to apply ID-preserving facial latent-space editing to faces which are out of the generator's domain. In this paper, we present an approach to bridge this gap. Our technique slightly alters the generator, so that an out-of-domain image is faithfully mapped into an in-domain latent code. The key idea is pivotal tuning -a brief training process that preserves the editing quality of an in-domain latent region, while changing its portrayed identity and appearance. In Pivotal Tuning Inversion (PTI), an initial inverted latent code serves as a pivot, around which the generator is finedtuned. At the same time, a regularization term keeps nearby identities intact, to locally contain the effect. This surgical training process ends up altering appearance features that represent mostly identity, without affecting editing capabilities. To supplement this, we further show that pivotal tuning can also adjust the generator to accommodate a multitude of faces, while introducing negligible distortion on the rest of the domain. We validate our technique through inversion and editing metrics, and show preferable scores to state-of-the-art methods. We further qualitatively demonstrate our technique by applying advanced edits (such as pose, age, or expression) to numerous images of well-known and recognizable identities. Finally, we demonstrate resilience to harder cases, including heavy make-up, elaborate hairstyles and/or headwear, which otherwise could not have been successfully inverted and edited by state-of-the-art methods.

HyperInverter: Improving StyleGAN Inversion via Hypernetwork

“…In addition, there are other two-stage approaches but are not hybrid ones. For example, PIE [44] uses optimization methods on both phases, while PTI [39] combines optimization process with the generator fine-tuning technique. In comparison, our work is also a two-stage method but differs from the above ones in that our approach is only based purely on the encoder-based manner in both phases.…”

“…Our goal in this phase is to recover the missing information of input x and thus reduce the discrepancy between x and xw . Other two-stage methods use either per-image optimization process [44] or per-image fine-tuning G [39] to improve reconstruction quality further. However, the drawback of such these approaches is the slow inference time.…”

“…PIE [44] opts first to use an optimization process to locate the latent code in W space to preserve the editing ability and then optimize further the latent in the W + space to enhance the reconstruction quality. PTI [39] approaches the same as PIE in the first stage. However, in the second stage, they choose the generator fine-tuning option to improve reconstruction results further.…”

SpaceEdit: Learning a Unified Editing Space for Open-Domain Image Editing

“…Although traversing the latent space of unconditional GANs can achieve image editing in closed-domain images such as faces, its incapability of generating real-world images (e.g., multiple objects and complex scenes) limits their generalization and application. In addition, since their hidden spaces need to retain all the information of the generated outputs, the inversion [54] of an open-domain image is usually compromised for photo fidelity [1,34]. In contrast, the editing space of our proposed model does not have such limitations.…”

“…One related work is styleGAN [19], which is trained to generate realistic images for closed-domain categories such as faces, cats, and cars. Since then, a series of manipulation works [6,34,35,41,44,45] have been built upon styleGAN by inverting a given image to its latent space and then manipulating the latent code to generate a new image while keeping the generator intact.…”