Grading the Severity of Mispronunciations in CAPT Based on Statistical Analysis and Computational Speech Perception

Abstract

Computer-aided pronunciation training (CAPT) technologies enable the use of automatic speech recognition to detect mispronunciations in second language (L2) learners' speech. In order to further facilitate learning, we aim to develop a principle-based method for generating a gradation of the severity of mispronunciations. This paper presents an approach towards gradation that is motivated by auditory perception. We have developed a computational method for generating a perceptual distance (PD) between two spoken phonemes. This is used to compute the auditory confusion of native language (L1). PD is found to correlate well with the mispronunciations detected in CAPT system for Chinese learners of English, i.e., L1 being Chinese (Mandarin and Cantonese) and L2 being US English. The results show that auditory confusion is indicative of pronunciation confusions in L2 learning. PD can also be used to help us grade the severity of errors (i.e., mispronunciations that confuse more distant phonemes are more severe) and accordingly prioritize the order of corrective feedback generated for the learners.

Visual-speech Synthesis of Exaggerated Corrective Feedback

“…By this method, we can provide more distinguishable feedback to learners and correct their mispronunciations. To synthesize emphatic speech, 8000 text prompts with phonelevel emphasis annotations are carefully designed, each containing one or more emphatic phones that the Chinese ESL speakers often mispronounce [9,17]. Two comparative speech utterances are recorded for each prompt: one with neutral intonation throughout the utterance and the other with exaggerated intonation with emphasis placed on the emphatic phones in the sentence.…”