Improving Mispronunciation Detection for Non-Native Learners with Multisource Information and LSTM-Based Deep Models

Li, Wei; Chen, Nancy F.; Siniscalchi, Sabato Marco; Lee, Chin‐Hui

doi:10.21437/interspeech.2017-464

Cited by 24 publications

(6 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Detection of mispronunciation has also been achieved by training a binary classifier to learn the decision boundary between correct and erroneous pronunciations. In [6,7], deep neural network (DNN) models have been applied to improve mispronunciation detection. Despite the high performance demonstrated, the training of DNN classifiers requires a large amount of disordered speech.…”

Section: Introductionmentioning

confidence: 99%

Child Speech Disorder Detection with Siamese Recurrent Network Using Speech Attribute Features

Wang¹,

Qin²,

Peng³

et al. 2019

Interspeech 2019

View full text Add to dashboard Cite

Acoustics-based automatic assessment is a highly desirable approach to detecting speech sound disorder (SSD) in children. The performance of an automatic speech assessment system depends greatly on the availability of a good amount of properly annotated disordered speech, which is a critical problem particularly for child speech. This paper presents a novel design of child speech disorder detection system that requires only normal speech for model training. The system is based on a Siamese recurrent network, which is trained to learn the similarity and discrepancy of pronunciations between a pair of phones in the embedding space. For detection of speech sound disorder, the trained network measures a distance that contrasts the test phone to the desired phone and the distance is used to train a binary classifier. Speech attribute features are incorporated to measure the pronunciation quality and provide diagnostic feedback. Experimental results show that Siamese recurrent network with a combination of speech attribute features and phone posterior features could attain an optimal detection accuracy of 0.941.

show abstract

Section: Introductionmentioning

confidence: 99%

Child Speech Disorder Detection with Siamese Recurrent Network Using Speech Attribute Features

Wang¹,

Qin²,

Peng³

et al. 2019

Interspeech 2019

View full text Add to dashboard Cite

show abstract

“…The LR classifier on top of the F-GOP algorithm outperformed the baseline due to the possibility to add informative features as input to the classifier. Mispronunciation at the phoneme level has been studied before using DNN models and Long Short Term Memory (LSTM) [10]. Various methods such as log-likelihood ratio [11] and GOP [12] were adopted previously.…”

Section: Related Workmentioning

confidence: 99%

Mispronunciation Detection for Spoken Isolated Words using Segmentation and Classification under Low Resource Conditions for Kannada Language

Murthy*,

Suresh,

Shah

et al. 2019

IJRTE

View full text Add to dashboard Cite

Relocation makes it inevitable for a person to learn the local pronunciations correctly. With the advent of mobile phones, language learning can be made easy and flexible. Our research involves Kannada Kali, a mobile and cloud based application that is being developed to assist people in learning the correct pronunciations of Kannada (a language spoken in India). Automatic Speech Recognition systems which are used to aid pronunciation training require to be trained on sufficient amount of spoken target language data. Since collecting such data in not easy, the objective of our research is to detect mispronounced segments of words with minimal data. When there is scarcity of data, a comparative approach where spoken word segments are compared with the canonical pronunciations is more effective for detecting anomaly in pronunciation. Since syllables are basic independent units of pronunciation, the spoken words are segmented into syllables for effective comparison and feedback. We propose an unsupervised segmentation method called Spectrogram Formant Contour Analysis that detects syllable boundaries by analysing the change in contours of the formants in the spoken word spectrograms. The task of mispronunciation detection is more effective when the application can identify the actual syllable pronounced and communicate the correct pronunciation to the user. For the purpose of syllable classification, our method employs a novel approach where a model is trained on phonemes and given syllables as input for identification. Our study includes comparing the performance of three machine learning algorithms, namely, Convolution Neural Network, Support Vector Machines and K-Nearest Neighbours on the task of identifying phonemes when they are trained on minimal data. The accuracy of KNN on phoneme classification was the best with 80% for clean and 60% for noisy data. In case of our initial results on syllable classification for Kannada Kali, Support Vector Machines gave the highest accuracy of almost 30%.

show abstract

“…It is well-known that the L2 learning process is heavily affected by a well-established habitual perception of phonemes and articulatory motions in the learners' primary language (L1) [1], which often cause mistakes and imprecise articulation in speech productions by the L2 learners, e.g., a negative language transfer [1,2]. As a feasible tool, computer assisted pronunciation training (CAPT) is often employed to automatically assess L2 learners' pronunciation quality at different levels, e.g., phonelevel [3][4][5][6][7][8][9][10][11][12], word-level [13][14][15][16][17] and sentence-level [18][19][20][21][22].…”

Section: Introductionmentioning

confidence: 99%

“…Alternatively, the two-step approaches treat pronunciation scoring or mispronunciation detection as regression or classification task. Specifically, phone, word and sentence boundaries are first generated by forced-alignment, and then either frame-level or segmental-level pronunciation features within each boundary are fed into task-dependent classifiers or regressors (e.g., [6,8,12,[15][16][17][18][19][20][21][22]). Finally, the posterior probabilities or predicted values obtained from those models are often used as pronunciation scores.…”

Section: Introductionmentioning

confidence: 99%

Improving Non-native Word-level Pronunciation Scoring with Phone-level Mixup Data Augmentation and Multi-source Information

Fu¹,

Gao²,

Wang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Deep learning-based pronunciation scoring models highly rely on the availability of the annotated non-native data, which is costly and has scalability issues. To deal with the data scarcity problem, data augmentation is commonly used for model pretraining. In this paper, we propose a phone-level mixup, a simple yet effective data augmentation method, to improve the performance of word-level pronunciation scoring. Specifically, given a phoneme sequence from lexicon, the artificial augmented word sample can be generated by randomly sampling from the corresponding phone-level features in training data, while the word score is the average of their GOP scores. Benefit from the arbitrary phone-level combination, the mixup is able to generate any word with various pronunciation scores. Moreover, we utilize multi-source information (e.g., MFCC and deep features) to further improve the scoring system performance. The experiments conducted on the Speechocean762 show that the proposed system outperforms the baseline by adding the mixup data for pretraining, with Pearson correlation coefficients (PCC) increasing from 0.567 to 0.61. The results also indicate that proposed method achieves similar performance by using 1/10 unlabeled data of baseline. In addition, the experimental results also demonstrate the efficiency of our proposed multisource approach.

show abstract

Improving Mispronunciation Detection for Non-Native Learners with Multisource Information and LSTM-Based Deep Models

Cited by 24 publications

References 21 publications

Child Speech Disorder Detection with Siamese Recurrent Network Using Speech Attribute Features

Child Speech Disorder Detection with Siamese Recurrent Network Using Speech Attribute Features

Mispronunciation Detection for Spoken Isolated Words using Segmentation and Classification under Low Resource Conditions for Kannada Language

Improving Non-native Word-level Pronunciation Scoring with Phone-level Mixup Data Augmentation and Multi-source Information

Contact Info

Product

Resources

About