Improving Multimodal Speech Enhancement by Incorporating Self-Supervised and Curriculum Learning

Abstract

Speech enhancement in realistic scenarios still remains many challenges, such as complex background signals and data limitations. In this paper, we present a co-attention based framework that incorporates self-supervised and curriculum learning to derive the target speech in noisy environments. Specifically, we first leverage self-supervision to pre-train the coattention model on the task of audio-visual synchronization.The pre-trained model can focus on the lip of speakers automatically, and then the self-supervised features from the model are combined with a u-net regression network to separate the spectrograms of sound mixtures. To make the training process easier and further improve the performance, we introduce the curriculum learning scheme for the training stage of speech enhancement. Extensive experiments show that our model achieves superior performance over previous selfsupervised method for speech enhancement, and demonstrate the generalizability of our approach to the transferred dataset.

RemixIT: Continual Self-Training of Speech Enhancement Models via Bootstrapped Remixing

“…However, in real-world settings, the speech recordings are degraded with additive noise, thus, self-learning robust embeddings becomes particularly challenging and demands the adaptation to the input noise distribution [20]. Several unsupervised speech denoising algorithms have been proposed by identifying and training with relatively clean segments of the noisy speech mixture [21], [22], using ASR losses [23], [24] exploiting visual cues [25], and harnessing the spatial separability of the sources using mic-arrays [26], [27]. Mixture invariant training (MixIT) [28] enables unsupervised training of separation models only with real-world singlechannel recordings by generating artificial mixtures of mixtures and estimating the independent sources.…”