Multilingual Adaptation of RNN Based ASR Systems

Miiller, Markus; Stiiker, Sebastian; Waibel, Alex

doi:10.1109/icassp.2018.8461614

Cited by 15 publications

(15 citation statements)

References 18 publications

(16 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although the universal multilingual modelling enjoys richer data resources, the mixture of data creates more variation, especially for those identical IPA symbols shared among different languages. The result is consistent with another recent independent study [29]. This motivates us to apply language adaptive training in the multilingual CTC model.…”

Section: Multilingual Trainingsupporting

confidence: 91%

Cross-lingual adaptation of a CTC-based multilingual acoustic model

Tong

Garner

Bourlard

2018

Speech Communication

View full text Add to dashboard Cite

Multilingual models for Automatic Speech Recognition (ASR) are attractive as they have been shown to benefit from more training data, and better lend themselves to adaptation to underresourced languages. However, initialisation from monolingual context-dependent models leads to an explosion of contextdependent states. Connectionist Temporal Classification (CTC) is a potential solution to this as it performs well with monophone labels.We investigate multilingual CTC in the context of adaptation and regularisation techniques that have been shown to be beneficial in more conventional contexts. The multilingual model is trained to model a universal International Phonetic Alphabet (IPA)-based phone set using the CTC loss function. Learning Hidden Unit Contribution (LHUC) is investigated to perform language adaptive training. In addition, dropout during cross-lingual adaptation is also studied and tested in order to mitigate the overfitting problem.Experiments show that the performance of the universal phoneme-based CTC system can be improved by applying LHUC and it is extensible to new phonemes during crosslingual adaptation. Updating all the parameters shows consistent improvement on limited data. Applying dropout during adaptation can further improve the system and achieve competitive performance with Deep Neural Network / Hidden Markov Model (DNN/HMM) systems on limited data.

show abstract

Section: Multilingual Trainingsupporting

confidence: 91%

Cross-lingual adaptation of a CTC-based multilingual acoustic model

Tong

Garner

Bourlard

2018

Speech Communication

View full text Add to dashboard Cite

show abstract

“…Thus, speaker features are typically added at the acoustic feature level. For language adaptation, we showed that adding LFVs deeper into the network [3] does improve the performance, and also optimized this approach [18]. Key is a method called modulation, which is based on Meta-PI networks [1,2].…”

Section: Lid Figure 1: Language Feature Vectors (Lfvs) Network Architmentioning

confidence: 99%

“…In this work, we will show how acoustic models can be adapted to languages in a multilingual setting. We have proposed adaptation methods based on language features [3,4] and now propose a new approach that extends our previous work. We will incorporate two novel aspects: a) the use of adaptive neural language codes (NLCs), which are based on language feature vectors (LFVs) [4], but can be adapted during acoustic model training and b) a network superstructure based on Meta-PI [1,2], which allows to use pre-trained monolingual subnets in a multilingual system.…”

Section: Introductionmentioning

confidence: 97%

Neural Language Codes for Multilingual Acoustic Models

2018

Self Cite

View full text Add to dashboard Cite

Multilingual Speech Recognition is one of the most costly AI problems, because each language (7,000+) and even different accents require their own acoustic models to obtain best recognition performance. Even though they all use the same phoneme symbols, each language and accent imposes its own coloring or "twang". Many adaptive approaches have been proposed, but they require further training, additional data and generally are inferior to monolingually trained models. In this paper, we propose a different approach that uses a large multilingual model that is modulated by the codes generated by an ancillary network that learns to code useful differences between the "twangs" or human language.We use Meta-Pi networks [1, 2] to have one network (the language code net) gate the activity of neurons in another (the acoustic model nets). Our results show that during recognition multilingual Meta-Pi networks quickly adapt to the proper language coloring without retraining or new data, and perform better than monolingually trained networks. The model was evaluated by training acoustic modeling nets and modulating language code nets jointly and optimize them for best recognition performance.

show abstract

“…One possible approach to this problem is to utilize data of other languages. There are various approaches to leverage other languages: (a) to train a model multilingually (multi-task learning with other languages), and then further fine-tune to a particular language [6], and (b) to adapt a multilingual model to a new language using transfer learning [6][7][8][9] and additional features obtained from the multilingual model such as multilingual bottleneck features (BNF) [10][11][12][13] and language feature vectors (LFV) [14] (cross-lingual adaptation). To obtain a multilingual S2S model, a part of parameters can be shared while preparing the output layers per language [6], and we can further use a unified architecture with a shared vocabulary among multiple languages [15][16][17].…”

Section: Introductionmentioning

confidence: 99%

Transfer Learning of Language-independent End-to-end ASR with Language Model Fusion

Inaguma

Cho

Baskar

et al. 2019

ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

This work explores better adaptation methods to low-resource languages using an external language model (LM) under the framework of transfer learning. We first build a language-independent ASR system in a unified sequence-to-sequence (S2S) architecture with a shared vocabulary among all languages. During adaptation, we perform LM fusion transfer, where an external LM is integrated into the decoder network of the attention-based S2S model in the whole adaptation stage, to effectively incorporate linguistic context of the target language. We also investigate various seed models for transfer learning. Experimental evaluations using the IARPA BA-BEL data set show that LM fusion transfer improves performances on all target five languages compared with simple transfer learning when the external text data is available. Our final system drastically reduces the performance gap from the hybrid systems.

show abstract

Multilingual Adaptation of RNN Based ASR Systems

Cited by 15 publications

References 18 publications

Cross-lingual adaptation of a CTC-based multilingual acoustic model

Cross-lingual adaptation of a CTC-based multilingual acoustic model

Neural Language Codes for Multilingual Acoustic Models

Transfer Learning of Language-independent End-to-end ASR with Language Model Fusion

Contact Info

Product

Resources

About