Cross-Lingual Phone Mapping for Large Vocabulary Speech Recognition of Under-Resourced Languages

Abstract

SUMMARYThis paper presents a novel acoustic modeling technique of large vocabulary automatic speech recognition for under-resourced languages by leveraging well-trained acoustic models of other languages (called source languages). The idea is to use source language acoustic model to score the acoustic features of the target language, and then map these scores to the posteriors of the target phones using a classifier. The target phone posteriors are then used for decoding in the usual way of hybrid acoustic modeling. The motivation of such a strategy is that human languages usually share similar phone sets and hence it may be easier to predict the target phone posteriors from the scores generated by source language acoustic models than to train from scratch an under-resourced language acoustic model. The proposed method is evaluated using on the Aurora-4 task with less than 1 hour of training data. Two types of source language acoustic models are considered, i.e. hybrid HMM/MLP and conventional HMM/GMM models. In addition, we also use triphone tied states in the mapping. Our experimental results show that by leveraging well trained Malay and Hungarian acoustic models, we achieved 9.0% word error rate (WER) given 55 minutes of English training data. This is close to the WER of 7.9% obtained by using the full 15 hours of training data and much better than the WER of 14.4% obtained by conventional acoustic modeling techniques with the same 55 minutes of training data.

Massively Multilingual Adversarial Speech Recognition

“…Multilingual transfer in ASR often relies on using bottle-neck features (Vesely et al, 2012;Vu et al, 2012;Karafiát et al, 2018) and adapting an acoustic model trained on one language to effectively recognize the sounds of other languages (Schultz and Waibel, 2001;Le and Besacier, 2005;Stolcke et al, 2006;Tóth et al, 2008;Plahl et al, 2011;Thomas et al, 2012;Imseng et al, 2014;Do et al, 2014;Heigold et al, 2013;Scharenborg et al, 2017). However, while most work uses less than 10 languages for model training, we include up to 100 languages in training.…”

Multilingual exemplar-based acoustic model for the NIST Open KWS 2015 evaluation

“…This makes a full fledged acoustic modeling process impractical for under-resourced languages. Popular approaches are to transfer well-trained acoustic models to under-resourced languages such as universal phone set [2,3], tandem approach [4][5][6], subspace GMMs (SGMMs) [7,8], Kullback-Leibler divergence HMM (KL-HMM) [9,10], crosslingual phone mapping [11][12][13] and its extension, contextdependent phone mapping [14][15][16]19].…”

Acoustic modeling for speech recognition under limited training data conditions

“…Transactions on Information and Systems [26], and in the two conferences: IALP 2012 [27] and ISCSLP 2012 [28]. The work on applying deep neural networks on monolingual speech recognition and cross-lingual phone mapping is published in the two conferences:…”

“…The results in this chapter have been published in: IEICE Transactions on Information and Systems [26], IALP 2012 [27], and ISCSLP 2012 [28].…”

“…The work also showed that the use of different source languages in combination can significantly improve the performance of phone mapping. This work has been published in the journal: IEICE Transactions on Information and Systems [26], and in the two conferences: IALP 2012 [27] and ISCSLP 2012 [28].…”