$$\text {Face2Face}^\rho $$: Real-Time High-Resolution One-Shot Face Reenactment

Yang, Kewei; Chen, Kang; Guo, Daoliang; Zhang, Song-Hai; Guo, Yuan-Chen; Zhang, Weidong

doi:10.1007/978-3-031-19778-9_4

Cited by 11 publications

(6 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To validate the performance of the proposed method, we compare our proposed face interactive coding scheme with the latest hybrid video coding standard VVC (VTM10.0) [3] and five generative compression schemes, including FOMM [16], FOMM2.0 [18], CFTE [24], Face vid2vid [5] and Face2FaceRHO [81]. We adopt three learning-based visual quality measures, including Deep Image Structure and Texture Similarity (DISTS) [94], Learned Perceptual Image Patch Similarity (LPIPS) [95] and Fréchet inception distance (FID) [96] to evaluate the reconstruction quality of generation results [97].…”

Section: Comparison Methodsmentioning

confidence: 99%

“…Yao et al [77] introduced graph convolutional networks (GCNs) to learn the optical flow from the reconstructed 3D meshes for oneshot face synthesis. In addition, Doukas et al [78], Zhang et al [79], Ren et al [80] and Yang et al [81] all leveraged the facial 3DMM in the controllable portrait generation from different strategies of multi-modality, multi-task learning or hierarchical motion prediction. Moreover, promising results in 3DMM-based face generative compression have also been shown in [82], [83].…”

Section: D Face Modeling With Deep Learningmentioning

confidence: 99%

“…Fig.7. Rate-distortion performance comparisons with VVC[3], FOMM[16], FOMM2.0[18], Face2FaceRHO[81], Face vid2vid[5], CFTE[24] in terms of DISTS, LPIPS and FID.…”

mentioning

confidence: 99%

“…Fig.10. Visual quality comparisons among VVC[3], FOMM[16], FOMM2.0[18], Face2FaceRHO[81], Face vid2vid[5], CFTE[24] and Ours at the similar bit rate. The values on the left represent coding bits and DISTS value, where lower DISTS value indicates better quality.…”

mentioning

confidence: 99%

See 3 more Smart Citations

Beyond Keypoint Coding: Temporal Evolution Inference with Compact Feature Representation for Talking Face Video Compression

Chen

Wang

et al. 2022

2022 Data Compression Conference (DCC)

View full text Add to dashboard Cite

In this paper, we propose a novel framework for Interactive Face Video Coding (IFVC), which allows humans to interact with the intrinsic visual representations instead of the signals. The proposed solution enjoys several distinct advantages, including ultra-compact representation, low delay interaction, and vivid expression and headpose animation. In particular, we propose the Internal Dimension Increase (IDI) based representation, greatly enhancing the fidelity and flexibility in rendering the appearance while maintaining reasonable representation cost. By leveraging strong statistical regularities, the visual signals can be effectively projected into controllable semantics in the three dimensional space (e.g., mouth motion, eye blinking, head rotation and head translation), which are compressed and transmitted. The editable bitstream, which naturally supports the interactivity at the semantic level, can synthesize the face frames via the strong inference ability of the deep generative model. Experimental results have demonstrated the performance superiority and application prospects of our proposed IFVC scheme. In particular, the proposed scheme not only outperforms the stateof-the-art video coding standard Versatile Video Coding (VVC) and the latest generative compression schemes in terms of rate-distortion performance for face videos, but also enables the interactive coding without introducing additional manipulation processes. Furthermore, the proposed framework is expected to shed lights on the future design of the digital human communication in the metaverse. The project page can be found at https://github.com/Berlin0610/Interactive Face Video Coding.

show abstract

Section: Comparison Methodsmentioning

confidence: 99%

Section: D Face Modeling With Deep Learningmentioning

confidence: 99%

“…Fig.7. Rate-distortion performance comparisons with VVC[3], FOMM[16], FOMM2.0[18], Face2FaceRHO[81], Face vid2vid[5], CFTE[24] in terms of DISTS, LPIPS and FID.…”

mentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

Beyond Keypoint Coding: Temporal Evolution Inference with Compact Feature Representation for Talking Face Video Compression

Chen

Wang

et al. 2022

2022 Data Compression Conference (DCC)

View full text Add to dashboard Cite

show abstract

“…However, such methods perform poorly on the challenging task of cross-subject reenactment, since facial landmarks preserve the facial shape and consequently the identity geometry of the target face. In order to mitigate the identity leakage from the target face to the source face, several methods [28,17,45,9,15,37,26,59] leverage the disentangled properties of 3D Morphable Models (3DMM) [8,16]. Xu et al [57] propose a unified architecture that learns to perform both face reenactment and swapping.…”

Section: Related Workmentioning

confidence: 99%

HyperReenact: One-Shot Reenactment via Jointly Learning to Refine and Retarget Faces

Bounareli,

Tzelepis,

Argyriou

et al. 2023

2023 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

OursFast BL FD PIR LSR LIA Rome Source TargetFigure 1: The proposed method, named HyperReenact, aims to synthesize realistic talking head sequences of a source identity driven by a target facial pose (i.e., 3D head orientation and facial expression). Our method performs both self and cross-subject reenactment and operates under the one-shot setting (i.e., using a single source frame). We demonstrate that the proposed framework can effectively reenact the source faces without producing significant visual artifacts, even on the challenging conditions of extreme head pose difference between the source and the target images (first row) and on crosssubject reenactment (second row). We compare our method against several state-of-the-art works on neural face reenactment, namely Fast BL [62], PIR [37], LSR [29], FD [9], LIA [53] and Rome [26].

show abstract

EDTalk: Efficient Disentanglement for Emotional Talking Head Synthesis

Tan,

Ji,

et al. 2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

$$\text {Face2Face}^\rho $$: Real-Time High-Resolution One-Shot Face Reenactment

Cited by 11 publications

References 47 publications

Beyond Keypoint Coding: Temporal Evolution Inference with Compact Feature Representation for Talking Face Video Compression

Beyond Keypoint Coding: Temporal Evolution Inference with Compact Feature Representation for Talking Face Video Compression

HyperReenact: One-Shot Reenactment via Jointly Learning to Refine and Retarget Faces

EDTalk: Efficient Disentanglement for Emotional Talking Head Synthesis

Contact Info

Product

Resources

About