Low Resource Comparison of Attention-based and Hybrid ASR Exploiting wav2vec 2.0

Abstract

Low resource speech recognition can potentially benefit a lot from exploiting a pretrained model such as wav2vec 2.0. These pretrained models have learned useful representations in an unsupervised or self-supervised task, often leveraging a very large corpus of untranscribed speech. The pretrained models can then be used in various ways. In this work we compare two approaches which exploit wav2vec 2.0: an attention-based end-to-end model (AED), where the wav2vec 2.0 model is used in the model encoder, and a hybrid hidden Markov model (HMM/DNN) speech recognition system, where the wav2vec 2.0 model is used in the acoustic model. These approaches are compared in a very difficult Northern Sámi task, as well as an easier, simulated low resource task in Finnish. We find that the wav2vec 2.0 AED models can learn a working attention mechanism, but are still outperformed by wav2vec 2.0 HMM/DNN systems. Our best wav2vec 2.0 HMM/DNN recipe on 20 hours is competitive with an HMM/DNN system trained on 1600 hours.

Speech Recognition System Improvements for North Sa ́mi Speaker-dependent and Speaker Independent Tasks

“…Previously, we applied wav2vec 2.0 pretrained Transformers to North Sámi [1]. We found that the hidden Markov model / deep neural network (HMM/DNN) approach outperformed the attention-based encoder-decoder (AED) approach, and thus are continuing our work here focusing on HMM/DNN-systems.…”

“…Here, we are able to leverage an additional North Sámi text resource. This resource, called Freecorpus (FC), consists of freely available texts, collected by Giellatekno and Divvun 1 .…”

“…We base our systems on the HMM-system recipe developed in [1]. This section presents the existing recipe.…”

Principled Comparisons for End-to-End Speech Recognition: Attention vs Hybrid at the 1000-Hour Scale

“…To the best of our knowledge, this proposed multihead inference is a novel improvement for the HMM/DNN approach, though it resembles an efficient form of model combination. We presented initial results using this approach in [44] and explore it here in more detail. Table I compares the various acoustic model training criteria and output heads used during inference.…”