wav2vec: Unsupervised Pre-Training for Speech Recognition

Abstract

We explore unsupervised pre-training for speech recognition by learning representations of raw audio. wav2vec is trained on large amounts of unlabeled audio data and the resulting representations are then used to improve acoustic model training. We pre-train a simple multi-layer convolutional neural network optimized via a noise contrastive binary classification task. Our experiments on WSJ reduce WER of a strong character-based log-mel filterbank baseline by up to 36 % when only a few hours of transcribed data is available. Our approach achieves 2.43 % WER on the nov92 test set. This outperforms Deep Speech 2, the best reported character-based system in the literature while using two orders of magnitude less labeled training data. 1

Unsupervised Pretraining Transfers Well Across Languages

“…We show the benefits of this by pre-training our modified CPC on 360 hours of unlabelled data from Librispeech and match the performance of the supervised model. This result not only confirms the findings of [6] but it also shows that unsupervised pre-training can match supervised pre-training with enough data (see Supplementary Section S2 with the larger Libri-light dataset [29]). In a second experiment, we compare the quality of our pre-trained features against other unsupervised methods on the Zerospeech2017.…”

“…However, CPC has the advantage of making no assumption about the nature or number of the training data samples. Recently, variants of CPC have been applied to monolingual ASR [6] and images [20].…”

“…Recently, several works [4,5] have proposed promising methods to train monolingual audio representations without supervision. In particular, Schneider et al [6] shows that the unsupervised pre-training method of van den Oord [4] improves the quality of automatic speech recognition (ASR) on several competitive benchmarks. In this paper, we are interested to see if similar unsupervised pre-training methods can be leveraged in a cross-lingual setting to improve the quality of phoneme representations for low resource languages.…”

“…We focus on the contrastive predictive coding (CPC) method of van den Oord [4] since Schneider et al [6] has shown its benefit for pre-training features for ASR. CPC is a form of forward modeling in the feature space [7]: it predicts the near future windows in an audio sequence while contrasting with windows from other sequences or more distant in time.…”

Unsupervised Pre-Training of Bidirectional Speech Encoders via Masked Reconstruction

“…In recent years, however, RBM-based pre-training has been largely abandoned, because direct supervised training of deep neural networks has improved due to new techniques such as better initialization [3], non-saturating activation functions [4], and better control of generalization [5]. However, very recent work has begun to reconsider the value of unsupervised pre-training, specifically in the context of representation learning on a large set of unlabeled data, for use in supervised training on a smaller set of labeled data [6,7,8].…”