A transfer learning approach to goodness of pronunciation based automatic mispronunciation detection

Huang, Hao; Xu, Haihua; Hu, Ying; Zhou, Gang

doi:10.1121/1.5011159

Cited by 23 publications

(11 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Pronunciation is a general term that includes several distinct features present in human languages. The correct pronunciation is hard and challenging to measure because there is no universal definition for correctness in the context of human languages, but there has been some research in this domain [5,6].…”

Section: Arabic Phonemes and Their Pronunciationmentioning

confidence: 99%

An Approach for Pronunciation Classification of Classical Arabic Phonemes Using Deep Learning

et al. 2021

View full text Add to dashboard Cite

A mispronunciation of Arabic short vowels can change the meaning of a complete sentence. For this reason, both the students and teachers of Classical Arabic (CA) are required extra practice for correcting students’ pronunciation of Arabic short vowels. That makes the teaching and learning task cumbersome for both parties. An intelligent process of students’ evaluation can make learning and teaching easier for both students and teachers. Given that online learning has become a norm these days, modern learning requires assessment by virtual teachers. In our case, the task is about recognizing the exact pronunciation of Arabic alphabets according to the standards. A major challenge in the recognition of precise pronunciation of Arabic alphabets is the correct identification of a large number of short vowels, which cannot be dealt with using traditional statistical audio processing techniques and machine learning models. Therefore, we developed a model that classifies Arabic short vowels using Deep Neural Networks (DNN). The model is constructed from scratch by: (i) collecting a new audio dataset, (ii) developing a neural network architecture, and (iii) optimizing and fine-tuning the developed model through several iterations to achieve high classification accuracy. Given a set of unseen audio samples of uttered short vowels, our proposed model has reached the testing accuracy of 95.77%. We can say that our results can be used by the experts and researchers for building better intelligent learning support systems in Arabic speech processing.

show abstract

Section: Arabic Phonemes and Their Pronunciationmentioning

confidence: 99%

An Approach for Pronunciation Classification of Classical Arabic Phonemes Using Deep Learning

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Recently, progress in deep learning for ASR [8] triggered new work on applying deep neural networks (DNNs) for pronunciation scoring [9,10,11,12,13], obtaining improvements over traditional methods for both the above-mentioned groups. Notably, methods of the second group usually rely on transfer learning approaches to mitigate the problem of data scarcity.…”

Section: Introductionmentioning

confidence: 99%

A transfer learning based approach for pronunciation scoring

Sancinetti¹,

Vidal²,

Bonomi³

et al. 2021

Preprint

View full text Add to dashboard Cite

Phone-level pronunciation scoring is a challenging task, with performance far from that of human annotators. Standard systems generate a score for each phone in a phrase using models trained for automatic speech recognition (ASR) with native data only. Better performance has been shown when using systems that are trained specifically for the task using nonnative data. Yet, such systems face the challenge that datasets labelled for this task are scarce and usually small. In this paper, we present a transfer learning-based approach that leverages a model trained for ASR, adapting it for the task of pronunciation scoring. We analyze the effect of several design choices and compare the performance with a state-of-the-art goodness of pronunciation (GOP) system. Our final system is 20% better than the GOP system on EpaDB, a database for pronunciation scoring research, for a cost function that prioritizes low rates of unnecessary corrections.

show abstract

“…However, the DNN-HMM models involved with subphonemic (senone) posterior probabilities and those cannot be mapped directly with HMM state transition probabilities because each senone is shared across many states [19]. Due to this, DNN-HMM based formulations introduced variants to the GoP without considering transition probabilities [17,[20][21][22]. Wenping et al computed the scores using senone posterior probability [17].…”

Section: Introductionmentioning

confidence: 99%

“…They also proposed another score by including the senone prior probabilities [20]. Further, the score as well as the features computed based on these score were used in the mispronunciation detection [23] considering transfer learning approach and pronunciation evaluation [22].…”

Section: Introductionmentioning

confidence: 99%

“…In the experimentation, we consider two native DNN-HMM acoustic models trained with the speech data from LibriSpeech (LS) [25] and Fisher-English (FE) [26] corpus. For the comparison, we consider three GoP formula- tions suggested in the work proposed by Wenping et al [17,20] and Huang et al [22] as the baseline, which do not include transition probabilities in their formulation. The correlation coefficients obtained with the proposed GoP formulations are found to be 0.637 and 0.409, which are 2.91% and 14.89% higher than the best among all the three baselines using the LS and FE based native models respectively.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

An Improved Goodness of Pronunciation (GoP) Measure for Pronunciation Evaluation with DNN-HMM System Considering HMM Transition Probabilities

Sudhakara¹,

Ramanathi²,

Yarra

et al. 2019

Interspeech 2019

View full text Add to dashboard Cite

Goodness of pronunciation (GoP) is typically formulated with Gaussian mixture model-hidden Markov model (GMM-HMM) based acoustic models considering HMM state transition probabilities (STPs) and GMM likelihoods of context dependent phonemes. On the other hand, deep neural network (DNN)-HMM based acoustic models employed sub-phonemic (senone) posteriors instead of GMM likelihoods along with STPs. However, each senone is shared across many states; thus, there is no one-to-one correspondence between them. In order to circumvent this, most of the existing works have proposed modifications to the GoP formulation considering only posteriors neglecting the STPs. In this work, we derive a formulation for the GoP and it results in the formulation involving both senone posteriors and STPs. Further, we illustrate the steps to implement the proposed GoP formulation in Kaldi, a state-of-the-art automatic speech recognition toolkit. Experiments are conducted on English data collected from Indian speakers using acoustic models trained with native English data from LibriSpeech and Fisher-English corpora. The highest improvement in the correlation coefficient between the scores from the formulations and the expert ratings is found to be 14.89% (relative) better with the proposed approach compared to the best of the existing formulations that don't include STPs.

show abstract

A transfer learning approach to goodness of pronunciation based automatic mispronunciation detection

Cited by 23 publications

References 34 publications

An Approach for Pronunciation Classification of Classical Arabic Phonemes Using Deep Learning

An Approach for Pronunciation Classification of Classical Arabic Phonemes Using Deep Learning

A transfer learning based approach for pronunciation scoring

An Improved Goodness of Pronunciation (GoP) Measure for Pronunciation Evaluation with DNN-HMM System Considering HMM Transition Probabilities

Contact Info

Product

Resources

About