Generative Pre-Training for Speech with Autoregressive Predictive Coding

Abstract

Learning meaningful and general representations from unannotated speech that are applicable to a wide range of tasks remains challenging. In this paper we propose to use autoregressive predictive coding (APC), a recently proposed self-supervised objective, as a generative pre-training approach for learning meaningful, non-specific, and transferable speech representations. We pre-train APC on largescale unlabeled data and conduct transfer learning experiments on three speech applications that require different information about speech characteristics to perform well: speech recognition, speech translation, and speaker identification. Extensive experiments show that APC not only outperforms surface features (e.g., log Mel spectrograms) and other popular representation learning methods on all three tasks, but is also effective at reducing downstream labeled data size and model parameters. We also investigate the use of Transformers for modeling APC and find it superior to RNNs.

Investigating Self-Supervised Pre-Training for End-to-End Speech Translation

“…Improvements of end-to-end AST were also proposed using weakly supervised data [21] or adding a second attention mechanism [22]. While supervised pretraining for AST was investigated (see for instance [16]), we are aware of a single research group [5,7] that investigated selfsupervised pre-training for AST. However their experiments were done in a high resource setting and AST (for which only marginal gains were displayed) was solely investigated among other tasks, without an in-depth analysis of the representations learnt.…”

“…As shown in Figure 1, we extract either wav2vec features or filter-bank+pitch features (later denoted as fbanks) from speech input. 5 Depending on the experiments, mean and variance normalization (MVN) is optionally applied to the generated features. For wav2vec feature extraction, we either use an off- 6 Data augmentation through speed perturbation is also applied with factors of 0.9, 1.0, and 1.1 to the training data.…”

“…Self-supervised learning using huge unlabeled data has been explored with very promising results for image processing [1] and natural language processing [2]. Recent works investigated selfsupervised representation learning from speech [3,4,5]. They were successful to improve performance on downstream tasks such as speech recognition.…”

“…An easier learning objective is introduced in Contrastive Predictive Coding (CPC) which consists in distinguishing a true future audio frame from negatives [3,8,9]. [5] shows that such representations are useful to improve several speech tasks while [4] extends those works by looking at the representations' robustness to domain and language shifts. In the same vein, [10] compares self-supervised and supervised pre-training for ASR and shows that CPC pre-training extracts features that transfer well to other languages, being on par or even outperforming supervised pretraining.…”

“…Practically each z i encodes 30ms of speech every 10ms. As for c i , the total receptive field of the context network is 210ms.3 https://github.com/pytorch/fairseq/blob/ master/examples/wav2vec/4 Note that these statistics were measured on our version of How2 downloaded on July 12, 2019[25] 5. Our preliminary experiments on How2 10% with MFCC features which lead to similar performance as filter-bank are not presented here.…”