A cappella: Audio-visual Singing Voice Separation

Montesinos, Juan F.; Kadandale, Venkatesh S.; Haro, Gloria

doi:10.48550/arxiv.2104.09946

Cited by 2 publications

(5 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…AV-CVAE [64] is an audio-visual speech separation model based on VAE [42], which detects the lip movements of the speaker and predicts the individual separated speech audio. Acapella [61] is for audio-visual singing separation. The architecture is a two-stream CNN, which is called Y-Net for processing audio and video respectively.…”

Section: Audio-visual Speech Separationmentioning

confidence: 99%

“…Lipreading WLAS [15],LiRA [56] Vid2Speech [24],Lipnet [3] Ephrat et al [25] Chung et al [15] AV speech separation AV-CVAE [64] VisualSpeech [29] Acapella [61],Facefilter [16] Ephrat et al [26] Afouras et al [1] Gabbay et al [27] Talking face generation DAVS [110],ATVGnet [8] Chung et al [14] Zhu et al [112] Song et al [86] Chen et al [7] Sound generation I2S [6],REGNET [9] Foley Music [28] Zhou et al [111] Figure 4: A taxonomy of audio-visual multimodal processing.…”

Section: Av Multimodal Processingmentioning

confidence: 99%

“…For instance, Vid2Speech [24] reported by Ephrat et al is able to handle lipreading task. Acapella [61] is a model for singing voice separation. VisualSpeech [29] is a versatile model, which achieves state of the art performance in speech separation and speech enhancement tasks.…”

mentioning

confidence: 99%

See 2 more Smart Citations

A Survey on Audio Synthesis and Audio-Visual Multimodal Processing

Shi

2021

Preprint

View full text Add to dashboard Cite

With the development of deep learning and artificial intelligence, audio synthesis has a pivotal role in the area of machine learning and shows strong applicability in the industry. Meanwhile, significant efforts have been dedicated by researchers to handle multimodal tasks at present such as audio-visual multimodal processing. In this paper, we conduct a survey on audio synthesis and audio-visual multimodal processing, which helps understand current research and future trends. This review focuses on text to speech(TTS), music generation and some tasks that combine visual and acoustic information. The corresponding technical methods are comprehensively classified and introduced, and their future development trends are prospected. This survey can provide some guidance for researchers who are interested in the areas like audio synthesis and audio-visual multimodal processing. introductionAudio synthesis, which aims to synthesis various form of natural and intelligible sound such as speech, music, has a wide range of application scenario in human society and industry. Initially, researchers took advantages of pure signal processing methods to find some convenient representations for audio, which can be easily modelled and transform to temporal audio. For example, short-time Fourier Transform(STFT) is an efficient way to convert audio into frequency domain and Griffin-Lim [31] is a kind of pure signal processing algorithm that is able to decode STFT sequence to temporal waveform. Methods similar to Griffin-Lim are WORLD [62], etc. In recent years, with the rapid development of deep learning technology, researchers have begun to build deep neural networks for audio synthesis and other multimodal tasks in order to simplify the pipeline and improve the performance of the model. Numerous neural network models have emerged so far for the tasks such as text to speech(TTS) and music generation. There are a lot of models for TTS that have been reported, such as Parallel WaveGAN [103], MelGAN [45], FastSpeech2/2s [80], EATs [21], VITS [40]. Simultaneously, there are many models for music generation like song from PI MuseGAN [23] and Jukebox [18]. These models bring great convenience to human production and life, and they provide key reference for future research.Vision is a physiological word. Humans and animals visually perceive the size, brightness, color, etc. of external objects, and obtain information that is essential for survival. Vision is the most important sense for human beings. Over these years, deep learning has been widely explored in various image processing and computer vision tasks such as image dehazing/deraining, objective detection and image segmentation, which contribute to the development of social productivity. Image dehazing/deraining means given a blurred image with haze/rain, algorithms are used to remove the haze/rain in the image to make it clear. [95,46,100,47,99,54,98,96,48] proposed neural network-based models for image dehazing/deraining respectively. Objective detection means finding out all ...

show abstract

Section: Audio-visual Speech Separationmentioning

confidence: 99%

Section: Av Multimodal Processingmentioning

confidence: 99%

See 1 more Smart Citation

A Survey on Audio Synthesis and Audio-Visual Multimodal Processing

Shi

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The results display that the approach can generalise well in real-world scenarios. Finally, Montesinos et al [16] proposed AV convolutional network based on graphs to separate singing voice by exploiting acoustic and visual information. The experimental results show the proposed approach outperforms state-of-the-art A-only singing voice separation approaches.…”

Section: Clean Imagesmentioning

confidence: 99%

“…In the literature, extensive studies have been carried out to develop AV SE methods in time-domain and frequency domain [4,5,6,7,8,9,10,11,12,13,14,15,16,17]. However, despite significant research in the area of AV SE, real-time processing models, with no or low latency (8-12 ms) remains a formidable technical challenge.…”

Section: Introductionmentioning

confidence: 99%

Towards Robust Real-time Audio-Visual Speech Enhancement

Gogate¹,

Dashtipour²,

Hussain³

2021

Preprint

View full text Add to dashboard Cite

The human brain contextually exploits heterogeneous sensory information to efficiently perform cognitive tasks including vision and hearing. For example, during the cocktail party situation, the human auditory cortex contextually integrates audio-visual (AV) cues in order to better perceive speech. Recent studies have shown that AV speech enhancement (SE) models can significantly improve speech quality and intelligibility in very low signal to noise ratio (SNR) environments as compared to audio-only SE models. However, despite significant research in the area of AV SE, development of real-time processing models with low latency remains a formidable technical challenge. In this paper, we present a novel framework for low latency speakerindependent AV SE that can generalise on a range of visual and acoustic noises. In particular, a generative adversarial networks (GAN) is proposed to address the practical issue of visual imperfections in AV SE. In addition, we propose a deep neural network based real-time AV SE model that takes into account the cleaned visual speech output from GAN to deliver more robust SE. The proposed framework is evaluated on synthetic and real noisy AV corpora using objective speech quality and intelligibility metrics and subjective listing tests. Comparative simulation results show that our real time AV SE framework outperforms state-of-the-art SE approaches, including recent DNN based SE models.

show abstract

A cappella: Audio-visual Singing Voice Separation

Cited by 2 publications

References 34 publications

A Survey on Audio Synthesis and Audio-Visual Multimodal Processing

A Survey on Audio Synthesis and Audio-Visual Multimodal Processing

Towards Robust Real-time Audio-Visual Speech Enhancement

Contact Info

Product

Resources

About