Lip-to-Speech Synthesis for Arbitrary Speakers in the Wild

Abstract

Figure 1: We address the problem of generating speech from silent lip videos for any speaker in the wild. Previous works train either on large amounts of data of isolated speakers or in laboratory settings with a limited vocabulary. Conversely, we can generate speech for the lip movements of arbitrary identities in any voice without additional speaker-specific fine-tuning. Our new VAE-GAN approach allows us to learn strong audio-visual associations despite the ambiguous nature of the task.

Towards Accurate Lip-to-Speech Synthesis in-the-Wild

“…Lip2Wav worked reasonably well on speakers seen during training but did not extend to handling unseen speakers. Several more recent studies [17,20,21,23] have attempted to develop end-to-end lipto-speech synthesis models on large datasets containing data from hundreds of speakers. For instance, in [17], authors proposed a variational approach that matches the distributions of lip movements and speech segments to project them into a shared space, which allows for handling the high variations of in-the-wild speakers to some extent.…”

“…Several more recent studies [17,20,21,23] have attempted to develop end-to-end lipto-speech synthesis models on large datasets containing data from hundreds of speakers. For instance, in [17], authors proposed a variational approach that matches the distributions of lip movements and speech segments to project them into a shared space, which allows for handling the high variations of in-the-wild speakers to some extent. Meanwhile, both [20,23] utilized a transformer-based approach to convert lip-to-speech synthesis into a sequence-tosequence problem, where a sequence of lip movements is translated into a sequence of speech tokens.…”

“…Comparisons: To evaluate lip-to-speech methods on the constrained single-speaker TCD-TIMIT dataset, we compare four existing approaches: (i) GAN-based [36], (ii) Lip2Wav [27], (iii) VAE-GAN [17], and (iv) VCA-GAN [20]. We adopt the same settings as Lip2Wav [27] and report the scores [27] and VCA-GAN [20].…”

“…Comparisons: In order to assess the performance of lip-tospeech methods in unconstrained scenarios, we employ three datasets: word-level LRW [8], sentence-level LRS2 [7], and LRS3 [3]. While the authors of VAE-GAN [17] have re-trained the GAN-based [36] and Lip2Wav [27] models in a multi-speaker context, we present the scores from their original study for comparison. For VCA-GAN [20], SVTS [23], and Multi-task Lip-to-Speech synthesis [21], we adopt the speech metric (PESQ, STOI and ESTOI) scores from [21].…”

“…While more recent efforts like [27] have extended lip-to-speech to "in-the-wild" environments, most of these models are speaker-specific in nature, i.e., they only work on speakers they are trained on. The speaker-independent models [17,20,21,23] suffer from numerous weaknesses and fail to accurately learn language and speech attributes like voice, prosody, etc. Due to the sheer difficulty of the task, they turn out to be well below expectations for an end-user application.…”