Generating Talking Face Landmarks from Speech

Abstract

The presence of a corresponding talking face has been shown to significantly improve speech intelligibility in noisy conditions and for hearing impaired population. In this paper, we present a system that can generate landmark points of a talking face from an acoustic speech in real time. The system uses a long short-term memory (LSTM) network and is trained on frontal videos of 27 different speakers with automatically extracted face landmarks. After training, it can produce talking face landmarks from the acoustic speech of unseen speakers and utterances. The training phase contains three key steps. We first transform landmarks of the first video frame to pin the two eye points into two predefined locations and apply the same transformation on all of the following video frames. We then remove the identity information by transforming the landmarks into a mean face shape across the entire training dataset. Finally, we train an LSTM network that takes the first-and second-order temporal differences of the log-mel spectrogram as input to predict face landmarks in each frame. We evaluate our system using the mean-squared error (MSE) loss of landmarks of lips between predicted and ground-truth landmarks as well as their first-and second-order temporal differences. We further evaluate our system by conducting subjective tests, where the subjects try to distinguish the real and fake videos of talking face landmarks. Both tests show promising results.

Hierarchical Cross-Modal Talking Face Generation With Dynamic Pixel-Wise Loss

“…Besides leveraging intermediate landmarks for avoiding directly correlating speech audio with irrelevant visual dynamics, we also propose a novel dynamically adjustable loss along with an attention mechanism to enforce the network to focus on audiovisual-correlated regions. It is worth to mention that in a recent audio-driven facial landmarks generation work [8], such irrelevant visual dynamics are removed in the training process by normalizing and identityremoving the facial landmarks. This has been shown to result in more natural synchronization between generated mouth shapes and speech audio.…”

Multi-modality Deep Restoration of Extremely Compressed Face Videos

“…Suwajanakorn et al [46] proposed an interesting technique to automatically edit a video of a given speaker with accurate lip synchronization guided by his own audio in a different speech. This work has spawned in recent years a number of variant methods on the task [47], [48], [49], [50], [51], [52], [53], [54], [55], [56], [57].…”

Speech Driven Talking Face Generation from a Single Image and an Emotion Condition

“…The presence of visual cues improves speech comprehension [1], [2], [3], [4] in noisy environments and for the hardof-hearing population. Consequently, researchers developed systems that can automatically generate talking faces from speech in order to provide the visual cues when they are not available [5], [6], [7], [8], [9], [10], [11], [12]. These systems can increase the accessibility of abundantly available audioonly resources for the hearing impaired population.…”