Autoregressive Co-Training for Learning Discrete Speech Representation

Learning Dependencies of Discrete Speech Representations with Neural Hidden Markov Models

“…To study the representations learned by our neural HMMs, we evaluate them on phone classification on Wall Street Journal (WSJ) and phone segmentation on TIMIT. We follow the same setting described in prior work [6,13,5], using Lib-riSpeech train-clean-360 for pre-training. Phone classification on WSJ is trained on 90% of si284, and evaluated on dev93.…”

“…Phone classification on WSJ is trained on 90% of si284, and evaluated on dev93. Following [5], we also evaluate phone cluster purity on WSJ si284. For phone segmentation on TIMIT, we do not follow the setting in other studies [24,22].…”

“…where W is a linear projection, V is the codebook, and d is the dimension of x t . We simply choose W the identity matrix, setting the codeword dimension to d. The choice of Gaussian is aligned with the L 2 loss in [5,14]. Note that the parametrization does not introduce any additional parameters compared to VQ-APC.…”

“…Discrete latent variable models for self-supervised learning [1,2,3,4] have shown strong performance in speech representation learning. Assuming discreteness in selfsupervised models aligns well with phonological categories in speech perception, and improves the accessibility of phonetic information [2,5]. Beyond discreteness, dependencies among discrete variables could be useful not only for speech recognition but also for speech segmentation and acoustic unit discovery.…”

“…Beyond discreteness, dependencies among discrete variables could be useful not only for speech recognition but also for speech segmentation and acoustic unit discovery. However, most self-supervised approaches [3,6,5] do not consider dependencies among discrete latent variables. In this work, we study the options of learning dependencies among latent variables in the context of selfsupervised learning.…”