Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis

Abstract

Humans involuntarily tend to infer parts of the conversation from lip movements when the speech is absent or corrupted by external noise. In this work, we explore the task of lip to speech synthesis, i.e., learning to generate natural speech given only the lip movements of a speaker. Acknowledging the importance of contextual and speakerspecific cues for accurate lip-reading, we take a different path from existing works. We focus on learning accurate lip sequences to speech mappings for individual speakers in unconstrained, large vocabulary settings. To this end, we collect and release a large-scale benchmark dataset, the first of its kind, specifically to train and evaluate the singlespeaker lip to speech task in natural settings. We propose a novel approach with key design choices to achieve accurate, natural lip to speech synthesis in such unconstrained scenarios for the first time. Extensive evaluation using quantitative, qualitative metrics and human evaluation shows that our method is four times more intelligible than previous works in this space.

Speech Prediction in Silent Videos Using Variational Autoencoders

“…In the following sections, we show quantitative comparisons based on the Short-Time Objective Intelligibility (STOI) [17] and Extended Short-Time Objective Intelligibility (ESTOI) [18] and Perceptual Evaluation of Speech Quality (PESQ) [19] metrics. We also make a qualitative comparison between samples from our model and current state-of-the-art Lip2Wav model [11]. As mentioned before, speech prediction is a many-to-many mapping problem.…”

“…Each sentence contains six words chosen from a fixed dictionary. Similar to prior works [7,9,11], we used four speakers (S1, S2, S4, and S29) from the dataset for comparison.…”

“…Figure 2 shows a qualitative comparison between our proposed model and currently state-of-the-art Lip2Wav model [11]. We found that the Lip2Wav model is biased towards overly smooth results that might explain the lower PESQ score, which measures speech quality, obtained by the Lip2Wav model.…”

“…al. [11] proposed a Tacotron2 model [12] variant for visual to speech generation. The model consists of a stack of 3D convolution layers that encode the frames' sequence into a fixed-size feature vector.…”

An Empirical Study of Visual Features for DNN Based Audio-Visual Speech Enhancement in Multi-Talker Environments

“…In [2], researchers have stated that viewing a speaker's face significantly enhances a person's capacity to understand the speech in a noisy environment. The use of visual modality has also been proved fruitful in different speech processing algorithms, such as audio visual speech recognition [3], lip reading [4,5], and lip to speech synthesis [6], etc. Recent studies also demonstrated that the use of visual features can help in speech denoising in very low signal to noise ratio (SNR) conditions [7,8].…”