TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech

Abstract

We introduce a self-supervised speech pre-training method called TERA, which stands for Transformer Encoder Representations from Alteration. Recent approaches often learn through the formulation of a single auxiliary task like contrastive prediction, autoregressive prediction, or masked reconstruction. Unlike previous approaches, we use a multi-target auxiliary task to pre-train Transformer Encoders on a large amount of unlabeled speech. The model learns through the reconstruction of acoustic frames from its altered counterpart, where we use a stochastic policy to alter along three dimensions: temporal, channel, and magnitude. TERA can be used to extract speech representations or fine-tune with downstream models. We evaluate TERA on several downstream tasks, including phoneme classification, speaker recognition, and speech recognition. TERA achieved strong performance on these tasks by improving upon surface features and outperforming previous methods. In our experiments, we show that through alteration along different dimensions, the model learns to encode distinct aspects of speech. We explore different knowledge transfer methods to incorporate the pre-trained model with downstream models. Furthermore, we show that the proposed method can be easily transferred to another dataset not used in pre-training.

What do End-to-End Speech Models Learn about Speaker, Language and Channel Information? A Layer-wise and Neuron-level Analysis

“…Deep Neural Networks have constantly pushed the state-of-the-art in speech technologies, for example automatic speech recognition (ASR) [2,3,4,5,6,7], pretrained speech transformers [8,9,10,11], dialect, language and speaker identification [12,13,14,15,16,17,18] models; along with other fields in Artificial Intelligence, including Natural Language Processing (NLP) [19] and Computer Vision (CV) [20]. While end-to-end deep architectures are simple, elegant and provide a flexible training mechanism, they are inherently black-boxes.…”

“…In this case, we used the official verification pairs to evaluate. 9 last accessed: April 10, 2020 10. Randomly selected ≈4 hours from each language.…”

Layer Reduction: Accelerating Conformer-Based Self-Supervised Model via Layer Consistency

“…Recently, self-supervised learning has shown great potential to empower a wide range of downstream tasks. For example, simSLR [2] in Computer Vision (CV) field provides comparable performance with supervised learning in image classification task; word and sentence representations learned from BERT [3], GPT [4] and their followers [5,6,7] maintain state-of-the-art results in multiple downstream Neural Language Processing (NLP) tasks; Speech representation extractors, like wav2vec [8,9] and TERA [10], provide more informative features and show significant performance improvement in downstream applications like Automatic Speech Recognition (ASR).…”

“…There are mainly three self-supervised learning paradigms in speech domain: Autoregressive Predictive Coding (APC) [11,12], Contrastive Predictive Coding (CPC) [13,8,9] and Masked Predictive Coding (MPC) [14,10], all of which try to encode semantic information (e.g., phonetic information) from contextual speech and output learned features for downstream tasks. Similar to autoregressive language model training in NLP domain, APC tries to predict future frames by encoding previous context in an autoregressive manner.…”

“…MPC, also known as BERT-like reconstruction method, is a training paradigm where bidirectional speech context is encoded by predicting continuously masked frame blocks. [10] is cited as a more comprehensive reference.…”