Acoustic Model Interpolation for Non-Native Speech Recognition

Abstract

This paper proposes three interpolation techniques which use the target language and the speaker's native language to improve nonnative speech recognition system. These interpolation techniques are manual interpolation, weighted least square and eigenvoices. Each of them can be used under different situation and constraints. In contrast to weighted least square and eigenvoices methods, manual interpolation can be achieved offline without any adaptation data. These methods can also be combined with MLLR to improve the recognition rate. Experiments presented in this paper show that the best non native adaptation method, combined with MLLR can give 10% WER absolute reduction on a French automatic speech recognition system for both Chinese and Vietnamese native speakers.

Non-Native Pronunciation Variation Modeling for Automatic Speech Recognition

“…In this subsection, we explain several interpolation methods, classified as either: 1) interpolation of native acoustic models of target language using non-native speech data (Steidl et al, 2004), and 2) interpolation of native acoustic models of target language based on native acoustic models of the mother tongue of non-native speakers (Tan et al, 2007).…”

“…However, in practice it is rather difficult to collect a sufficient amount of non-native speech; therefore, acoustic models are usually adapted via a conventional acoustic model adaptation method, such as maximum likelihood linear regression (MLLR) and/or maximum a posteriori (MAP) methods (Yang et al, 2004). As an alternative, the acoustic models adjusted for non-native speech can also be obtained by interpolating the acoustic models for native speech and the acoustic models for the mother tongue (Steidl et al, 2004;Tan et al, 2007). In other words, the acoustic models trained with two different languages are combined to obtain the acoustic models for non-native speech.…”

“…In other words, acoustic or pronunciation modeling approaches can be combined in an MLLR and/or MAP adaptation framework (Goronzy et al, 2004;He et al, 2003;Liu et al, 2008;Oh et al, 2007;Tan et al, 2007). In particular, Bouselmi et al (Bouselmi et al, 2007) proposed several combination schemes for pronunciation and MLLR/MAP acoustic model adaptations.…”

“…As an effort to adapt acoustic models, several interpolation methods have been proposed, where two sets of acoustic models, the acoustic models trained with the target language and the acoustic models trained with the mother tongue of native speakers, are combined (Matsunaga et al, 2003;Steidl et al, 2004;Tan et al, 2007). Contrary to interpolating acoustic models, phone acoustic models of the target language were modified by adding an alternative path to the corresponding mother tongue phone acoustic models of non-native speakers (Bartkova et al, 2006;Bouselmi et al, 2006).…”

“…Tan and Besacier (Tan et al, 2007) proposed three interpolation methods based on the use of both the target language acoustic models and the mother tongue acoustic models of nonnative speakers, which include 1) manual interpolation, 2) weighted least square based interpolation, and 3) eigenvoice based interpolation. The three acoustic model interpolation methods consist of two identical steps for preprocessing and one different step for the acoustic model interpolation.…”