Are discrete units necessary for Spoken Language Modeling?

Abstract

Recent work in spoken language modeling shows the possibility of learning a language unsupervisedly from raw audio without any text labels. The approach relies first on transforming the audio into a sequence of discrete units (or pseudo-text) and then training a language model directly on such pseudo-text. Is such a discrete bottleneck necessary, potentially introducing irreversible errors in the encoding of the speech signal, or could we learn a language model without discrete units at all? In this work, show that discretization is indeed essential for good results in spoken language modeling, but that can omit the discrete bottleneck if we use using discrete target features from a higher level than the input features. We also show that an end-to-end model trained with discrete target like HuBERT achieves similar results as the best language model trained on pseudo-text on a set of zero-shot spoken language modeling metrics from the Zero Resource Speech Challenge 2021.

Variable-rate hierarchical CPC leads to acoustic unit discovery in speech

“…, u q T : u q k = q(u k ). Quantizing the targets has been shown to improve the quality of speech representations in contrastive prediction-based self-supervised learning [Baevski et al, 2020, Nguyen et al, 2022, perhaps by contributing to the prior of discreteness we wish to impose on the high-level features.…”