Speech Driven Talking Face Generation from a Single Image and an Emotion Condition

Abstract

Visual emotion expression plays an important role in audiovisual speech communication. In this work, we propose a novel approach to rendering visual emotion expression in speech-driven talking face generation. Specifically, we design an end-to-end talking face generation system that takes a speech utterance, a single face image, and a categorical emotion label as input to render a talking face video in sync with the speech and expressing the condition emotion. Objective evaluation on image quality, audiovisual synchronization, and visual emotion expression shows that the proposed system outperforms a stateof-the-art baseline system. Subjective evaluation of visual emotion expression and video realness also demonstrates the superiority of the proposed system. Furthermore, we conduct a pilot study on human emotion recognition of generated videos with mismatched emotions between the audio and visual modalities, and results show that humans reply on the visual modality more significantly than the audio modality on this task.

Investigating Contributions of Speech and Facial Landmarks for Talking Head Generation

“…Another recent end-to-end system for talking face generation from noisy speech has been studied with image quality and mouth-shape synchronization, which is attained by a mouth region mask loss (MRM) [8]. In their following work [9], an end-to-end talking face generation system receives a reference face image, a speech utterance, and a categorical emotion label to generate a talking face video in sync with the speech and expressing the conditioned emotion. They discard the synchronization discriminator from their previous work and keep only the MRM loss for the mouth movements.…”