Transformer Based End-to-End Mispronunciation Detection and Diagnosis

Wu, Minglin; Li, Kun; Leung, Wai-Kim; Meng, Helen

doi:10.21437/interspeech.2021-1467

Cited by 22 publications

(12 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Goodness-of-pronunciation (GOP) is among the first DNN-based methods to MDD, which relies on phone posterior outputs from an automatic speech recognizer (ASR) [1,2,23] to evaluate phonetic errors. More recently, end-to-end phoneme recognition has been studied [5,3,4,8,24], among which [4] and [24] also explored fine-tuning Wav2vec 2.0. Our proposed method differs with them in that we investigate the usage of unlabeled target domain speech to enhance MDD performance.…”

Section: Related Workmentioning

confidence: 99%

“…We compute per-speaker PERs between the recognized phonemes and the canonical phonemes given by a grapheme-to-phoneme model. 3 Then we select 10 highest-PER speakers and 10 lowest-PER speakers, and randomly sample 10 utterances for each of the 20 speakers. 17 human listeners score the accentedness (scale: 1-9 where 1 means heavy accent) and intelligibility (scale: 0-100 where 0 means not intelligible at all) of the sampled utterances.…”

Section: Open Test: Indian Accent and Intelligibility Assessmentmentioning

confidence: 99%

“…Second-language (L2) English learners typically present accents and mispronunciations, which highly impact their intelligibility in practical communication. In recent years, Computer Aided Pronunciation Training (CAPT) tools have been developed to provide diagnosis and feedback on phonetic-level errors (phoneme substitution, deletion, insertion [1,2,3,4,5]) and prosodic-level errors (e.g. lexical stress, intonation [6]).…”

Section: Introductionmentioning

confidence: 99%

“…Currently, most phonetic level mispronunciation detection and diagnosis (MDD) systems perform end-to-end phoneme recognition on L2 speech, based on deep neural network (DNN) architectures [3,4,5,7,8]. One of the challenges of training such DNNs is data sparsity, due to the laborious process of annotating perceived phonemes on L2 speech.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Improving Mispronunciation Detection with Wav2vec2-based Momentum Pseudo-Labeling for Accentedness and Intelligibility Assessment

Yang¹,

Hirschi²,

Looney³

et al. 2022

Preprint

View full text Add to dashboard Cite

Current leading mispronunciation detection and diagnosis (MDD) systems achieve promising performance via end-to-end phoneme recognition. One challenge of such end-to-end solutions is the scarcity of human-annotated phonemes on natural L2 speech. In this work, we leverage unlabeled L2 speech via a pseudo-labeling (PL) procedure and extend the fine-tuning approach based on pre-trained self-supervised learning (SSL) models. Specifically, we use Wav2vec 2.0 as our SSL model, and fine-tune it using original labeled L2 speech samples plus the created pseudo-labeled L2 speech samples. Our pseudo labels are dynamic and are produced by an ensemble of the online model on-the-fly, which ensures that our model is robust to pseudo label noise. We show that fine-tuning with pseudo labels gains a 5.35% phoneme error rate reduction and 2.48% MDD F1 score improvement over a labeled-samples-only fine-tuning baseline. The proposed PL method is also shown to outperform conventional offline PL methods. Compared to the state-of-theart MDD systems, our MDD solution achieves a more accurate and consistent phonetic error diagnosis. In addition, we conduct an open test on a separate UTD-4Accents dataset, where our system recognition outputs show a strong correlation with human perception, based on accentedness and intelligibility.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Open Test: Indian Accent and Intelligibility Assessmentmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Improving Mispronunciation Detection with Wav2vec2-based Momentum Pseudo-Labeling for Accentedness and Intelligibility Assessment

Yang¹,

Hirschi²,

Looney³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The computer-assisted pronunciation training (CAPT) system, which can conduct assessments and provide detailed feedback on pronunciation proficiency, is thus attracting attention as an ESL learning service and platform [1,2]. There are two technical approaches to the CAPT system: mispronunciation detection and diagnosis (MDD) [3,4,5,6,7,8,9,10,11,12] and automatic pronunciation assessment [6,13,14,15,16,17,10]. MDD is a task of detecting pronunciation errors by calculating multiple measures using estimated and canonical phones from an automatic speech recognizer.…”

Section: Introductionmentioning

confidence: 99%

Automatic Pronunciation Assessment using Self-Supervised Speech Representation Learning

Kim¹,

Jeon²,

Seo³

et al. 2022

Preprint

View full text Add to dashboard Cite

Self-supervised learning (SSL) approaches such as wav2vec 2.0 and HuBERT models have shown promising results in various downstream tasks in the speech community. In particular, speech representations learned by SSL models have been shown to be effective for encoding various speech-related characteristics. In this context, we propose a novel automatic pronunciation assessment method based on SSL models. First, the proposed method fine-tunes the pre-trained SSL models with connectionist temporal classification to adapt the English pronunciation of English-as-a-second-language (ESL) learners in a data environment. Then, the layer-wise contextual representations are extracted from all across the transformer layers of the SSL models. Finally, the automatic pronunciation score is estimated using bidirectional long short-term memory with the layer-wise contextual representations and the corresponding text. We show that the proposed SSL model-based methods outperform the baselines, in terms of the Pearson correlation coefficient, on datasets of Korean ESL learner children and Speechocean762. Furthermore, we analyze how different representations of transformer layers in the SSL model affect the performance of the pronunciation assessment task.

show abstract

Mispronunciation detection and diagnosis using deep neural networks: a systematic review

Lounis,

Dendani,

Bahi

2024

Multimed Tools Appl

View full text Add to dashboard Cite

Transformer Based End-to-End Mispronunciation Detection and Diagnosis

Cited by 22 publications

References 19 publications

Improving Mispronunciation Detection with Wav2vec2-based Momentum Pseudo-Labeling for Accentedness and Intelligibility Assessment

Improving Mispronunciation Detection with Wav2vec2-based Momentum Pseudo-Labeling for Accentedness and Intelligibility Assessment

Automatic Pronunciation Assessment using Self-Supervised Speech Representation Learning

Mispronunciation detection and diagnosis using deep neural networks: a systematic review

Contact Info

Product

Resources

About