Unsupervised Pre-Training of Bidirectional Speech Encoders via Masked Reconstruction

Abstract

We propose an approach for pre-training speech representations via a masked reconstruction loss. Our pre-trained encoder networks are bidirectional and can therefore be used directly in typical bidirectional speech recognition models. The pre-trained networks can then be fine-tuned on a smaller amount of supervised data for speech recognition. Experiments with this approach on the LibriSpeech and Wall Street Journal corpora show promising results. We find that the main factors that lead to speech recognition improvements are: masking segments of sufficient width in both time and frequency, pre-training on a much larger amount of unlabeled data than the labeled data, and domain adaptation when the unlabeled and labeled data come from different domains. The gain from pre-training is additive to that of supervised data augmentation.

Data Augmenting Contrastive Learning of Speech Representations in the Time Domain

“…CPC2 is a modified version of the CPC architecture in [2,3]. The encoder architecture is unchanged (5 convolutional layers with kernel sizes [10,8,4,4,4], strides [5,4,2,2,2] and hidden dimension 256). We increase the depth of the auto-regressive network, which improves accuracy (see Supplementary S1) For the recurrent context nextwork, we use a 2-layer LSTM, as a tradeoff between feature quality and training speed.…”

Towards Unsupervised Learning of Speech Features in the Wild

“…In this paper, we use a re-implementation of the CPC model [34], which we call CPC2. The encoder architecture is the same (5 convolutional layers with kernel sizes [10,8,4,4,4], strides [5,4,2,2,2] and hidden dimension 256), for the context network, we used 2-layer LSTM, and for the prediction network, we used a multi-head transformer [35], each of the 12 heads predicting one future time slice.…”

“…Unsupervised representation learning has been studied as a topic of its own [1,2,3], but recently gained attention as a pretraining method to obtain speech features that can be fine tuned for downstream application with little labelled data [4,5,6,7,8]. This opens up the prospect of constructing speech technology for low resource languages.…”

Lattice-Free Mmi Adaptation of Self-Supervised Pretrained Acoustic Models

“…This approach is referred to as semi-supervised learning. More recently, self-supervised learning methods that transform the input signal to learn powerful representations have been receiving a lot of attention [10,11,12,13,14,15]. In contrast to semi-supervised learning, self-supervised learning aims to improve the seed model by exploiting unlabelled data before adaptation on supervised data.…”

“…Self-supervised training approaches can be broadly grouped into two classes: (1) auto-regressive models that try to predict the future representations conditional on the past inputs [13,11] and (2) bidirectional models that learn to predict masked parts of the input [10,15,12]. In [12,10], authors explore the adaptation of a bidirectional network for ASR.…”