Video-Driven Speech Reconstruction using Generative Adversarial Networks

Vougioukas, Konstantinos; Ma, P.; Petridis, Stavros; Pantić, Maja

doi:10.48550/arxiv.1906.06301

Cited by 10 publications

(27 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…2) Speaker-independent Result: for speaker-independent cases, we follow the same setups for GRID [34] and TCD-TIMIT [32].…”

Section: Resultsmentioning

confidence: 99%

“…Kumar et al [33] validate the effectiveness of using multiple views of faces on both speaker-dependent and -independent speech reconstruction. Vougioukas et al [34] utilize generative adversarial networks (GAN) to directly predict raw waveforms from visual inputs in an end-to-end fashion without generating an intermediate representation of audios. Inspired by the speech synthesis model, Tacotron2 [35], Qu et al propose to directly map video inputs to low-level speech representations, mel-spectrogram, with an encoder-decoder architecture and achieve better results on lip reading experiments.…”

Section: A Lip To Speech Reconstructionmentioning

confidence: 99%

See 1 more Smart Citation

LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading

Qu,

Weber,

Wermter

2021

Preprint

View full text Add to dashboard Cite

The aim of this work is to investigate the impact of crossmodal self-supervised pre-training for speech reconstruction (video-to-audio) by leveraging the natural co-occurrence of audio and visual streams in videos. We propose LipSound2 which consists of an encoder-decoder architecture and locationaware attention mechanism to map face image sequences to mel-scale spectrograms directly without requiring any human annotations. The proposed LipSound2 model is firstly pre-trained on ∼2400h multi-lingual (e.g. English and German) audiovisual data (VoxCeleb2). To verify the generalizability of the proposed method, we then fine-tune the pre-trained model on domain-specific datasets (GRID, TCD-TIMIT) for English speech reconstruction and achieve a significant improvement on speech quality and intelligibility compared to previous approaches in speaker-dependent and -independent settings. In addition to English, we conduct Chinese speech reconstruction on the CMLR dataset to verify the impact on transferability. Lastly, we train the cascaded lip reading (video-to-text) system by fine-tuning the generated audios on a pre-trained speech recognition system and achieve state-of-the-art performance on both English and Chinese benchmark datasets.

show abstract

“…2) Speaker-independent Result: for speaker-independent cases, we follow the same setups for GRID [34] and TCD-TIMIT [32].…”

Section: Resultsmentioning

confidence: 99%

Section: A Lip To Speech Reconstructionmentioning

confidence: 99%

LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading

Qu,

Weber,

Wermter

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…While this can be effective for the GRID corpus that has no head movements, optical flow could be a detrimental feature in unconstrained settings due to large head pose changes. Another work [36] strives for improved speech quality by generating raw waveforms using GANs. However, both these works do not make use of the well-studied sequence-to-sequence paradigm [31] that is used for text-to-speech generation [30]; thus leaving a large room for improvement in speech quality and correctness.…”

Section: Lip To Speech Generationmentioning

confidence: 99%

“…Prior works in lip to speech regard their speech representation as a 2D-image [10,36] in the case of melspectrograms or as a single feature vector [10] in the case of LPC features. In both these cases, they use a 2D-CNN to decode these speech representations.…”

Section: Problem Formulationmentioning

confidence: 99%

“…The LPC features are low-dimensional and easier to generate, however, they result in robotic, artificial sounding speech. At the other extreme [36], one can generate raw waveforms but the high dimensionality of the output (16000 samples per second) makes the network training process computationally inefficient. We take inspiration from previous text-to-speech works [27,30] and generate melspectrograms conditioned on lip movements.…”

Section: Speech Representationmentioning

confidence: 99%

See 1 more Smart Citation

Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis

Prajwal

Mukhopadhyay

Namboodiri

et al. 2020

Preprint

View full text Add to dashboard Cite

Humans involuntarily tend to infer parts of the conversation from lip movements when the speech is absent or corrupted by external noise. In this work, we explore the task of lip to speech synthesis, i.e., learning to generate natural speech given only the lip movements of a speaker. Acknowledging the importance of contextual and speakerspecific cues for accurate lip-reading, we take a different path from existing works. We focus on learning accurate lip sequences to speech mappings for individual speakers in unconstrained, large vocabulary settings. To this end, we collect and release a large-scale benchmark dataset, the first of its kind, specifically to train and evaluate the singlespeaker lip to speech task in natural settings. We propose a novel approach with key design choices to achieve accurate, natural lip to speech synthesis in such unconstrained scenarios for the first time. Extensive evaluation using quantitative, qualitative metrics and human evaluation shows that our method is four times more intelligible than previous works in this space. Please check out our demo video for a quick overview of the paper, method, and qualitative results. https://www.youtube.com/watch? v=HziA-jmlk_4&feature=youtu.be

show abstract

Fairness-based Bidding Model for Renewable Portfolio Standard Policy in South Korea Standard Policy in South Korea

Kim

Seok

2022

J. Electr. Eng. Technol.

View full text Add to dashboard Cite

Recent studies have shown impressive performance in Lip-tospeech synthesis that aims to reconstruct speech from visual information alone. However, they have been suffering from synthesizing accurate speech in the wild, due to insufficient supervision for guiding the model to infer the correct content. Distinct from the previous methods, in this paper, we develop a powerful Lip2Speech method that can reconstruct speech with correct contents from the input lip movements, even in a wild environment. To this end, we design multitask learning that guides the model using multimodal supervision, i.e. text and audio, to complement the insufficient word representations of acoustic feature reconstruction loss. Thus, the proposed framework brings the advantage of synthesizing speech containing the right content of multiple speakers with unconstrained sentences. We verify the effectiveness of the proposed method using LRS2, LRS3, and LRW datasets.

show abstract

Video-Driven Speech Reconstruction using Generative Adversarial Networks

Cited by 10 publications

References 0 publications

LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading

LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading

Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis

Fairness-based Bidding Model for Renewable Portfolio Standard Policy in South Korea Standard Policy in South Korea

Contact Info

Product

Resources

About