DPE: Disentanglement of Pose and Expression for General Video Portrait Editing

Pang, Youxin; Zhang, Yong; Quan, Weize; Fan, Yanbo; Cun, Xiaodong; Shi, Ying; Yan, Dong‐Ming

doi:10.48550/arxiv.2301.06281

Cited by 1 publication

(3 citation statements)

References 29 publications

(49 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Expression Loss L exp . Previous methods have used facial expression recognition network (Pang et al 2023) to calculate the expression loss. However, on one hand, the performance of facial expression recognition is instability.…”

Section: Loss Functionmentioning

confidence: 99%

“…To evaluate the effectiveness of our disentanglement mechanism, we compare our approach with two SOTA disentanglementbased talking head generation methods, i.e., StyleHeat (Yin et al 2022) and DPE (Pang et al 2023). As shown in Tab.…”

Section: Disentanglement Of Pose and Expressionmentioning

confidence: 99%

“…This task is useful yet challenging due to two entangled issues: 1) isolating expression-specific features for more finegrained granular control, and 2) providing AU-based control and alignment of human faces with expressions. The most related works (Ren et al 2021;Yin et al 2022;Pang et al 2023) adopt feature disentanglement mechanisms to extract pose and expression from input videos. However, we have found these pose and expression features also contain appearance information in our pilot study.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

FG-EmoTalk: Talking Head Video Generation with Fine-Grained Controllable Facial Expressions

Sun,

Xuan,

Liu

et al. 2024

AAAI

View full text Add to dashboard Cite

Although deep generative models have greatly improved one-shot video-driven talking head generation, few studies address fine-grained controllable facial expression editing, which is crucial for practical applications. Existing methods rely on a fixed set of predefined discrete emotion labels or simply copy expressions from input videos. This is limiting as expressions are complex, and methods using only emotion labels cannot generate fine-grained, accurate or mixed expressions. Generating talking head video with precise expressions is also difficult using 3D model-based approaches, as 3DMM only models facial movements and tends to produce deviations. In this paper, we propose a novel framework enabling fine-grained facial expression editing in talking face generation. Our goal is to achieve expression control by manipulating the intensities of individual facial Action Units (AUs) or groups. First, compared with existing methods which decouple the face into pose and expression, we propose a disentanglement scheme to isolates three components from the human face, namely, appearance, pose, and expression. Second, we propose to use input AUs to control muscle group intensities in the generated face, and integrate the AUs features with the disentangled expression latent code. Finally, we present a self-supervised training strategy with well-designed constraints. Experiments show our method achieves fine-grained expression control, produces high-quality talking head videos and outperforms baseline methods.

show abstract

Section: Loss Functionmentioning

confidence: 99%

Section: Disentanglement Of Pose and Expressionmentioning

confidence: 99%