Mispronunciation Detection Leveraging Maximum Performance Criterion Training of Acoustic Models and Decision Functions

Hsu, Yao Chi; Yang, Ming; Hung, Hsiao Tsung; Chen, Berlin

doi:10.21437/interspeech.2016-1602

Cited by 12 publications

(6 citation statements)

References 22 publications

(25 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A direct comparison of our F1 score with existing work is not possible due to the difference in audio data characteristics. However, as a rough comparison, the current state-of-the art on miscue detection tasks [7,8] report an F1 score of about 0.6 to 0.65, which is comparable to our score of 0.819 on augmented data. However, for real data we obtained a slightly lower F1 score of 0.512.…”

Section: Discussionsupporting

confidence: 81%

“…To create artificial data with miscues, we inspected existing literature on the different types of miscues. Unlike existing works on binary miscue detection task [7,8], we need miscue types that occur in natural speech of reading. Since our goal is to classify miscue types for reading corpus at the phoneme level, we selected miscue types [9] that are applicable at the phoneme level.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Automatic Miscue Detection Using RNN Based Models with Data Augmentation

Hong¹,

Gweon

2018

Interspeech 2018

View full text Add to dashboard Cite

This study proposes a method of using data augmentation to address the problem of data shortages in miscue detection tasks. Three main steps were taken. First, a phoneme classifier was developed to acquire force-aligned data, which would be used for miscue classification and data augmentation. In order to create the phoneme classifier, phonetic features of "Seoul Reading Speech" (SRS) corpus were extracted by using grapheme-to-phoneme (G2P) to train CNN-based models. Second, to obtain miscue labeled corpus, we performed data augmentation using the phoneme classifier output, which is artificially generated miscue corpus of SRS (modified-SRS). This miscue corpus was created by randomly deleting or modifying sound sections according to three miscue categories; extension (EXT), pause (PAU), and pre-correction (PRE). Third, the performance of the miscue classifier was tested after training three types of RNN based models (LSTM, BiLSTM, BiGRU) with the modified-SRS corpus. The results show that the BiGRU model performed best at 0.819 in F1-score on augmented data, while BiLSTM model performed best at 0.512 on real data.

show abstract

Section: Discussionsupporting

confidence: 81%

Section: Introductionmentioning

confidence: 99%

Automatic Miscue Detection Using RNN Based Models with Data Augmentation

Hong¹,

Gweon

2018

Interspeech 2018

View full text Add to dashboard Cite

show abstract

“…The paper uses phoneme-level statistics and also does prosody analysis by prosody feature extraction. The work in [16] estimates acoustic models by leveraging deep neural network. All the above methods mentioned above require sufficient amount of speech recordings for training the models for efficient mispronunciation detection.…”

Section: Related Workmentioning

confidence: 99%

Mispronunciation Detection for Spoken Isolated Words using Segmentation and Classification under Low Resource Conditions for Kannada Language

Murthy*,

Suresh,

Shah

et al. 2019

IJRTE

View full text Add to dashboard Cite

Relocation makes it inevitable for a person to learn the local pronunciations correctly. With the advent of mobile phones, language learning can be made easy and flexible. Our research involves Kannada Kali, a mobile and cloud based application that is being developed to assist people in learning the correct pronunciations of Kannada (a language spoken in India). Automatic Speech Recognition systems which are used to aid pronunciation training require to be trained on sufficient amount of spoken target language data. Since collecting such data in not easy, the objective of our research is to detect mispronounced segments of words with minimal data. When there is scarcity of data, a comparative approach where spoken word segments are compared with the canonical pronunciations is more effective for detecting anomaly in pronunciation. Since syllables are basic independent units of pronunciation, the spoken words are segmented into syllables for effective comparison and feedback. We propose an unsupervised segmentation method called Spectrogram Formant Contour Analysis that detects syllable boundaries by analysing the change in contours of the formants in the spoken word spectrograms. The task of mispronunciation detection is more effective when the application can identify the actual syllable pronounced and communicate the correct pronunciation to the user. For the purpose of syllable classification, our method employs a novel approach where a model is trained on phonemes and given syllables as input for identification. Our study includes comparing the performance of three machine learning algorithms, namely, Convolution Neural Network, Support Vector Machines and K-Nearest Neighbours on the task of identifying phonemes when they are trained on minimal data. The accuracy of KNN on phoneme classification was the best with 80% for clean and 60% for noisy data. In case of our initial results on syllable classification for Kannada Kali, Support Vector Machines gave the highest accuracy of almost 30%.

show abstract

“…In Eqs. (30) and (31), the occupation-data, sum-of-data, and sum-of-square-data of the "numerator" are b…”

Section: Transfer Learning Based Gop Using Gmm-hmmsmentioning

confidence: 99%

A transfer learning approach to goodness of pronunciation based automatic mispronunciation detection

Huang

et al. 2017

The Journal of the Acoustical Society of America

View full text Add to dashboard Cite

Goodness of pronunciation (GOP) is the most widely used method for automatic mispronunciation detection. In this paper, a transfer learning approach to GOP based mispronunciation detection when applying maximum F1-score criterion (MFC) training to deep neural network (DNN)-hidden Markov model based acoustic models is proposed. Rather than train the whole network using MFC, a DNN is used, whose hidden layers are borrowed from native speech recognition with only the softmax layer trained according to the MFC objective function. As a result, significant mispronunciation detection improvement is obtained. In light of this, the two-stage transfer learning based GOP is investigated in depth. The first stage exploits the hidden layer(s) to extract phonetic-discriminating features. The second stage uses a trainable softmax layer to learn the human standard for judgment. The validation is carried out by experimenting with different mispronunciation detection architectures using acoustic models trained by different criteria. It is found that it is preferable to use frame-level cross-entropy to train the hidden layer parameters. Classifier based mispronunciation detection is further experimented with using features computed by transfer learning based GOP and it is shown that it also helps to achieve better results.

show abstract

Mispronunciation Detection Leveraging Maximum Performance Criterion Training of Acoustic Models and Decision Functions

Cited by 12 publications

References 22 publications

Automatic Miscue Detection Using RNN Based Models with Data Augmentation

Automatic Miscue Detection Using RNN Based Models with Data Augmentation

Mispronunciation Detection for Spoken Isolated Words using Segmentation and Classification under Low Resource Conditions for Kannada Language

A transfer learning approach to goodness of pronunciation based automatic mispronunciation detection

Contact Info

Product

Resources

About