HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing

Alaluf, Yuval; Tov, Omer; Mokady, Ron; Gal, Ran; Bermano, Amit H.

doi:10.1109/cvpr52688.2022.01796

Cited by 155 publications

(70 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent works have focused mostly on the inversion of the popular StyleGAN, building on previous work with a variety of inversion structures and minimization objectives [59][60][61][62][63] with the aim of generalization to any dataset. However, we used a simpler and narrow approach by developing our own StyleGAN inverter for the W space using a naive recoding network.…”

Section: Gan Inversionmentioning

confidence: 99%

Comparing the latent space of generative models

Asperti

Valerio

2022

Neural Comput & Applic

View full text Add to dashboard Cite

Different encodings of datapoints in the latent space of latent-vector generative models may result in more or less effective and disentangled characterizations of the different explanatory factors of variation behind the data. Many works have been recently devoted to the exploration of the latent space of specific models, mostly focused on the study of how features are disentangled and of how trajectories producing desired alterations of data in the visible space can be found. In this work we address the more general problem of comparing the latent spaces of different models, looking for transformations between them. We confined the investigation to the familiar and largely investigated case of generative models for the data manifold of human faces. The surprising, preliminary result reported in this article is that (provided models have not been taught or explicitly conceived to act differently) a simple linear mapping is enough to pass from a latent space to another while preserving most of the information. This is full of consequences for representation learning, potentially paving the way to the transformation of editing trajectories from one space to another, or the adaptation of disentanglement techniques between different generative domains.

show abstract

Section: Gan Inversionmentioning

confidence: 99%

Comparing the latent space of generative models

Asperti

Valerio

2022

Neural Comput & Applic

View full text Add to dashboard Cite

show abstract

“…MSE, perceptual similarity metric using LPIPS [53] and structural similarity metric using MS-SSIM [49]. In addition, for facial reconstruction, we follow recent 2D GAN inversion works [13,4] and measure identity similarity using a pre-trained facial recognition network of CurricularFace [22]. Furthermore, we also need to measure the 3D reconstruction ability of our method, as the clear advantage of 3D GAN inversion over 2D GAN inversion is that the former allows for novel view synthesis given a single input image.…”

Section: Reconstructionmentioning

confidence: 99%

3D GAN Inversion with Pose Optimization

Ko¹,

Cho²,

Choi³

et al. 2022

Preprint

View full text Add to dashboard Cite

With the recent advances in NeRF-based 3D aware GANs quality, projecting an image into the latent space of these 3D-aware GANs has a natural advantage over 2D GAN inversion: not only does it allow multi-view consistent editing of the projected image, but it also enables 3D reconstruction and novel view synthesis when given only a single image. However, the explicit viewpoint control acts as a main hindrance in the 3D GAN inversion process, as both camera pose and latent code have to be optimized simultaneously to reconstruct the given image. Most works that explore the latent space of the 3D-aware GANs rely on ground-truth camera viewpoint or deformable 3D model, thus limiting their applicability. In this work, we introduce a generalizable 3D GAN inversion method that infers camera viewpoint and latent code simultaneously to enable multi-view consistent semantic image editing. The key to our approach is to leverage pre-trained estimators for better initialization and utilize the pixel-wise depth calculated from NeRF parameters to better reconstruct the given image.We conduct extensive experiments on image reconstruction and editing both quantitatively and qualitatively, and further compare our results with 2D GAN-based editing to demonstrate the advantages of utilizing the latent space of 3D GANs. Additional results and visualizations are available at 3dgan-inversion. github.io

show abstract

“…These methods provide high-quality reconstruction but also require heavy computation. In contrast to optimization-based methods, learning-based methods allow lighter and faster reconstruction by training an encoder to directly output the latent code [29,2], or by training a hypernetwork for defining a refined generator to improve reconstruction quality [3,9].…”

Section: Related Workmentioning

confidence: 99%

“…A common approach to perform data editing with a generative adversarial network is to follow a two-stage principle: invert first, edit later. This principle has been well demonstrated in the 2D domain, where state-of-the-art GAN inversion methods [29,2,3,9] are used to map a real image into the latent space of the StyleGAN model [20] by using optimization-based or encoder-based techniques. Once the mapping is done, the reconstructed latent code can be manipulated to make effects on the real image (e.g., changing attributes of the image).…”

Section: Introductionmentioning

confidence: 99%

PointInverter: Point Cloud Reconstruction and Editing via a Generative Model with Shape Priors

Kim¹,

Hua²,

Nguyen³

et al. 2022

Preprint

View full text Add to dashboard Cite

In this paper, we propose a new method for mapping a 3D point cloud to the latent space of a 3D generative adversarial network. Our generative model for 3D point clouds is based on SP-GAN, a state-of-the-art sphere-guided 3D point cloud generator. We derive an efficient way to encode an input 3D point cloud to the latent space of the SP-GAN. Our point cloud encoder can resolve the point ordering issue during inversion, and thus can determine the correspondences between points in the generated 3D point cloud and those in the canonical sphere used by the generator. We show that our method outperforms previous GAN inversion methods for 3D point clouds, achieving state-of-the-art results both quantitatively and qualitatively. Our code is available at https: //github.com/hkust-vgd/point_inverter.

show abstract

HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing

Cited by 155 publications

References 26 publications

Comparing the latent space of generative models

Comparing the latent space of generative models

3D GAN Inversion with Pose Optimization

PointInverter: Point Cloud Reconstruction and Editing via a Generative Model with Shape Priors

Contact Info

Product

Resources

About