An Investigation of Deep Neural Networks for Multilingual Speech Recognition Training and Adaptation

Abstract

Different training and adaptation techniques for multilingual Automatic Speech Recognition (ASR) are explored in the context of hybrid systems, exploiting Deep Neural Networks (DNN) and Hidden Markov Models (HMM). In multilingual DNN training, the hidden layers (possibly extracting bottleneck features) are usually shared across languages, and the output layer can either model multiple sets of language-specific senones or one single universal IPA-based multilingual senone set. Both architectures are investigated, exploiting and comparing different language adaptive training (LAT) techniques originating from successful DNN-based speaker-adaptation. More specifically, speaker adaptive training methods such as Cluster Adaptive Training (CAT) and Learning Hidden Unit Contribution (LHUC) are considered. In addition, a language adaptive output architecture for IPA-based universal DNN is also studied and tested.Experiments show that LAT improves the performance and adaptation on the top layer further improves the accuracy. By combining state-level minimum Bayes risk (sMBR) sequence training with LAT, we show that a language adaptively trained IPA-based universal DNN outperforms a monolingually sequence trained model.

Multilingual Sequence-to-Sequence Speech Recognition: Architecture, Transfer Learning, and Language Modeling

“…To alleviate the limitation in the previous approach, the final layer of the seq2seq model which is mainly responsible for classification is retrained to the target language. In previous works [11,30] related to hybrid DNN/RNN models and CTC based models [12,15] the softmax layer is only adapted. However in our case, the attention decoder and CTC decoder both have to be retrained to the target language.…”

An Investigation of Multilingual ASR Using End-to-end LF-MMI

“…We hypothesize that biphone targets cover more variabilities compared to the corresponding monophone, especially when they are shared by mutiliple languages. As reported in [20], language-specific characteristics cannot be well modeled by an IPA-based universal network. Language adaptive training (LAT) could be a solution to better model these variabilities across languages.…”

“…It has been demonstrated that the layers close to the output layer are more language-related and training the last layer in a language-dependent manner can help the IPA-based multilingual system to better capture the language specificity [20]. More specifically, the output of the neural network for language s, o sL , is calculated as…”

Multilingual Bottleneck Features for Query by Example Spoken Term Detection

“…The outputs are mono-phone based tied states (also known as pdfs in Kaldi [23]) corresponding to each language as presented in Section 5.1. The training labels for these networks are generated using GMM-HMM based speech recognizers [24,25]. The number of classes corresponding to French, German, Portuguese, Spanish and Russian are 124, 133, 145, 130, 151 respectively.…”