Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependencies

Abstract

Self-supervised speech representations have been shown to be effective in a variety of speech applications. However, existing representation learning methods generally rely on the autoregressive model and/or observed global dependencies while generating the representation. In this work, we propose Non-Autoregressive Predictive Coding (NPC), a self-supervised method, to learn a speech representation in a non-autoregressive manner by relying only on local dependencies of speech. NPC has a conceptually simple objective and can be implemented easily with the introduced Masked Convolution Blocks. NPC offers a significant speedup for inference since it is parallelizable in time and has a fixed inference time for each time step regardless of the input sequence length. We discuss and verify the effectiveness of NPC by theoretically and empirically comparing it with other methods. We show that the NPC representation is comparable to other methods in speech experiments on phonetic and speaker classification while being more efficient.

Self-Supervised Speech Representation Learning: A Review

“…Some studies explore alternatives to masking the input directly. In non-autoregressive predictive coding (NPC) [89], time masking is introduced through masked convolution blocks. Taking inspiration from XLNet [90], it has also been suggested that the input be reconstructed from a shuffled version [91] to address the discrepancy between pre-training and fine-tuning of masking-based approaches.…”

Audio Self-supervised Learning: A Survey

“…Transformer encoders and bidirectional RNNs have been considered as context networks for realising MPC. Similarly, the recently proposed Non-autoregressive predictive coding (NPC) [52] also applies a mask on its model input, but it learns representations based on local dependencies of an input sequence, rather than globally. The MPC approaches can learn effective representations of sequential data in a non-autoregressive way, and hence achieve considerable speed-up in training.…”

A Comparative Study of Self-Supervised Speech Representation Based Voice Conversion

“…Generative modeling incorporates language model-like training losses to predict unseen regions (such as future or masked frames), in order to maximize the likelihood of the observed data. Examples include APC [43], VQ-APC [44], Mockingjay [45], TERA [46], and NPC [47]. Discriminative modeling aims to discriminate (or contrast) the target unseen frame with randomly sampled ones, which is equivalent to mutual information maximization.…”