StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGAN

Yin, Fang‐Fang; Zhang, Yong; Cun, Xiaodong; Cao, Mingdeng; Fan, Yanbo; Wang, Xuan; Bai, Qingyan; Wang, Jue; Yang, Yujiu

doi:10.48550/arxiv.2203.04036

Cited by 6 publications

(13 citation statements)

References 60 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The model presented in [45] modularizes audio-visual representations by devising an implicit low-dimension pose code to tackle the problem of rhythmic head motion. In StyleHEAT [44], the authors show how to utilize StyleGAN [17] model to create talking faces guided by speech embeddings but also controlled by intuitive or attribute editing. Some modern approaches utilize rendering networks to obtain more accurate face 3D representation.…”

Section: Related Workmentioning

confidence: 99%

“…Generated heads move and behave in a natural expressive way while still preserving the subject's identity and plausible lip sync. In contrast to most recent approaches [3,15,24,29,35,40,[44][45][46], we use Denoising Diffusion Probabilistic Models [12,19] that utilize a variational approach instead of adversarial training and do not require stabilizing multiple discriminators. To eliminate the problem of unnaturally-looking sequences, we introduce motion frames (see Section 4.2) that are recurrently guiding video creation.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation

Stypułkowski¹,

Vougioukas²,

He³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation

Stypułkowski¹,

Vougioukas²,

He³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The former maps a randomly sampled noise to a style latent code of 512 dimensions while the latter produces satisfying images with this latent code and a constant input by Adaptive Instance Normalization layers. To deal with conditional synthesis tasks, recent methods [5,8,9,10,11,12] use a technique called GAN inversion [13]. GAN inversion is to map an image into the latent space of a pretrained GAN model for a desired latent code, which can be faithfully reconstructed afterwards.…”

Section: Preliminariesmentioning

confidence: 99%

Identity-Guided Face Generation with Multi-Modal Contour Conditions

Bai

Xia

Yin

et al. 2022

2022 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

Recent face generation methods have tried to synthesize faces based on the given contour condition, like a low-resolution image or sketch. However, the problem of identity ambiguity remains unsolved, which usually occurs when the contour is too vague to provide reliable identity information (e.g., when its resolution is extremely low). Thus feasible solutions of image restoration could be infinite. In this work, we propose a novel framework that takes the contour and an extra image specifying the identity as the inputs, where the contour can be of various modalities, including the low-resolution image, sketch, and semantic label map. Concretely, we propose a novel dual-encoder architecture, in which an identity encoder extracts the identity-related feature, accompanied by a main encoder to obtain the rough contour information and further fuse all the information together. The encoder output is iteratively fed into a pre-trained StyleGAN generator until getting a satisfying result. To the best of our knowledge, this is the first work that achieves identity-guided face generation conditioned on multi-modal contour images. Moreover, our method can produce photo-realistic results with 1024×1024 resolution.

show abstract

“…Face editing via GAN (Generative Adversarial Network) inversion [1] enables users to flexibly edit a wide range of facial attributes in real face images. Existing methods [2,3,4,5] first invert face images into the latent space of 2D GANs such as StyleGAN [6], then manipulate the style codes, and finally feed the edited codes into the pre-trained generator to obtain the edited face images. However, 2D GANs lack the knowledge of the underlying 3D structure of the faces, and their 3D consistency in multi-view generation is limited, as shown in Fig.…”

Section: Introductionmentioning

confidence: 99%

Designing a 3D-Aware StyleNeRF Encoder for Face Editing

Yang¹,

Wang²,

Peng³

et al. 2023

Preprint

View full text Add to dashboard Cite

GAN inversion has been exploited in many face manipulation tasks, but 2D GANs often fail to generate multi-view 3D consistent images. The encoders designed for 2D GANs are not able to provide sufficient 3D information for the inversion and editing. Therefore, 3Daware GAN inversion is proposed to increase the 3D editing capability of GANs. However, the 3D-aware GAN inversion remains underexplored. To tackle this problem, we propose a 3D-aware (3Da) encoder for GAN inversion and face editing based on the powerful StyleNeRF model. Our proposed 3Da encoder combines a parametric 3D face model with a learnable detail representation model to generate geometry, texture and view direction codes. For more flexible face manipulation, we then design a dual-branch StyleFlow module to transfer the StyleNeRF codes with disentangled geometry and texture flows. Extensive experiments demonstrate that we realize 3D consistent face manipulation in both facial attribute editing and texture transfer. Furthermore, for video editing, we make the sequence of frame codes share a common canonical manifold, which improves the temporal consistency of the edited attributes.

show abstract

StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGAN

Cited by 6 publications

References 60 publications

Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation

Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation

Identity-Guided Face Generation with Multi-Modal Contour Conditions

Designing a 3D-Aware StyleNeRF Encoder for Face Editing

Contact Info

Product

Resources

About