Learning to Dub Movies via Hierarchical Prosody Models

Cong, Gaoxiang; Li, Liang; Qi, Yuankai; Zha, Zheng-Jun; Wu, Qi; Wang, Wenyu; Jiang, Bin; Yang, Ming-Hsuan; Huang, Qingming

doi:10.1109/cvpr52729.2023.01411

Cited by 3 publications

(1 citation statement)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Based on the findings, most studies did not specify the language used for lip synchronization [1], [6]- [8], [10], [13], [17], [18], [27]- [29], [32], [33], [35], [36]. A reasonable assumption is that studies that do not specify a language will likely utilize English, given its status as the universal language of scientific discourse.…”

Section: B Rq2: Spoken Languages Used For Synchronizationmentioning

confidence: 99%

A Systematic Literature Review: Facial Expression and Lip Movement Synchronization of an Audio Track

Hassan Alshahrani,

Maashi

2024

IEEE Access

View full text Add to dashboard Cite

This systematic literature review (SLR) explores the topic of Facial Expression and Lip Movement Synchronization of an Audio Track in the context of Automatic Dubbing. This SLR aims to gain insights into the advancements, trends, and limitations of technologies related to this topic. The review protocol was well-defined, and we formulated specific research questions to guide the review process. A comprehensive search strategy included reputable electronic databases and relevant keywords. The study selection process followed inclusion and exclusion criteria, including 32 relevant studies published from 1995 to 2024. We systematically performed data extraction to gather critical information from the selected studies. The findings of this SLR contribute to understanding the current state-of-the-art technologies, identifying research gaps, and informing further advancements in this field.

show abstract

Section: B Rq2: Spoken Languages Used For Synchronizationmentioning

confidence: 99%

A Systematic Literature Review: Facial Expression and Lip Movement Synchronization of an Audio Track

Hassan Alshahrani,

Maashi

2024

IEEE Access

View full text Add to dashboard Cite

show abstract

RobustL2S: Speaker-Specific Lip-to-Speech Synthesis exploiting Self-Supervised Representations

Sahipjohn,

Shah,

Tambrahalli

et al. 2023

2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)

View full text Add to dashboard Cite

Fast and Accurate Facial Expression Image Classification and Regression Method Based on Knowledge Distillation

2023

View full text Add to dashboard Cite

As emotional states are diverse, simply classifying them through discrete facial expressions has its limitations. Therefore, to create a facial expression recognition system for practical applications, not only must facial expressions be classified, emotional changes must be measured as continuous values. Based on the knowledge distillation structure and the teacher-bounded loss function, we propose a method to maximize the synergistic effect of jointly learning discrete and continuous emotional states of eight expression classes, valences, and arousal levels. The proposed knowledge distillation model uses Emonet, a state-of-the-art continuous estimation method, as the teacher model, and uses a lightweight network as the student model. It was confirmed that performance degradation can be minimized even though student models have multiply-accumulate operations of approximately 3.9 G and 0.3 G when using EfficientFormer and MobileNetV2, respectively, which is much less than the amount of computation required by the teacher model (16.99 G). Together with the significant improvements in computational efficiency (by 4.35 and 56.63 times using EfficientFormer and MobileNetV2, respectively), the decreases in facial expression classification accuracy were approximately 1.35% and 1.64%, respectively. Therefore, the proposed method is optimized for application-level interaction systems in terms of both the amount of computation required and the accuracy.

show abstract

Learning to Dub Movies via Hierarchical Prosody Models

Cited by 3 publications

References 39 publications

A Systematic Literature Review: Facial Expression and Lip Movement Synchronization of an Audio Track

A Systematic Literature Review: Facial Expression and Lip Movement Synchronization of an Audio Track

RobustL2S: Speaker-Specific Lip-to-Speech Synthesis exploiting Self-Supervised Representations

Fast and Accurate Facial Expression Image Classification and Regression Method Based on Knowledge Distillation

Contact Info

Product

Resources

About