Multi-Task Self-Supervised Learning for Robust Speech Recognition

Abstract

Despite the growing interest in unsupervised learning, extracting meaningful knowledge from unlabelled audio remains an open challenge. To take a step in this direction, we recently proposed a problem-agnostic speech encoder (PASE), that combines a convolutional encoder followed by multiple neural networks, called workers, tasked to solve self-supervised problems (i.e., ones that do not require manual annotations as ground truth). PASE was shown to capture relevant speech information, including speaker voice-print and phonemes. This paper proposes PASE+, an improved version of PASE for robust speech recognition in noisy and reverberant environments. To this end, we employ an online speech distortion module, that contaminates the input signals with a variety of random disturbances. We then propose a revised encoder that better learns short-and long-term speech dynamics with an efficient combination of recurrent and convolutional networks. Finally, we refine the set of workers used in self-supervision to encourage better cooperation.Results on TIMIT, DIRHA and CHiME-5 show that PASE+ significantly outperforms both the previous version of PASE as well as common acoustic features. Interestingly, PASE+ learns transferable representations suitable for highly mismatched acoustic conditions.

Data Augmenting Contrastive Learning of Speech Representations in the Time Domain

“…Further work will allow to determine whether the superiority of time domain noise augmentation over spectral ones is specific to the CPC loss or to the fact that our architecture starts directly from the waveform as opposed to using spectral features like Mel Filterbanks or MFCCs. Note that [18] also combines several data augmentation techniques for unsupervised learning in an autoencoder architecture. Among data augmentation technique they use the most are two time-domain ones (reverberation and additive noise) and one spectral (band reject).…”

“…Our work is close to [18], which applies data augmentation techniques to representation learning (autoencoders). However, they evaluated them in terms of pretraining for a downstream task not in terms of the learned representation.…”

Adaptation Algorithms for Neural Network-Based Speech Recognition: An Overview

“…Such deep generative models offer different ways of addressing the problem of adaptation including powerful approaches to data augmentation, and the development of rich adaptation algorithms building on a base model with a joint distribution over acoustics and symbols. This offers the possibility of finetuning general encoders to specific acoustic domains, and adapting the decoder to specific tasks (such as speech recognition, speaker identification, language recognition, or emotion recognition), noting that classic adaptation to speakers can bring further gains [327], [328].…”

Self-Supervised Learning From Multi-Sensor Data for Sleep Recognition

“…Self-supervised learning has been firstly adopted within the computer vision community to learn representations by solving various auxiliary tasks, such as colorize gray scale images or solving puzzles from image patches. Self-supervised learning has also been applied successfully in language modeling, leading to models like BERT [25]. In this problem, the self-supervised learning model can increase the data capacity very well, so that the model can be trained using unlabeled data.…”