Mispronunciation Detection in Non-Native (L2) English with Uncertainty Modeling

Abstract

A common approach to the automatic detection of mispronunciation in language learning is to recognize the phonemes produced by a student and compare it to the expected pronunciation of a native speaker. This approach makes two simplifying assumptions: a) phonemes can be recognized from speech with high accuracy, b) there is a single correct way for a sentence to be pronounced. These assumptions do not always hold, which can result in a significant amount of false mispronunciation alarms. We propose a novel approach to overcome this problem based on two principles: a) taking into account uncertainty in the automatic phoneme recognition step, b) accounting for the fact that there may be multiple valid pronunciations. We evaluate the model on non-native (L2) English speech of German, Italian and Polish speakers, where it is shown to increase the precision of detecting mispronunciations by up to 18% (relative) compared to the common approach.

Computer-assisted Pronunciation Training -- Speech synthesis is almost all you need

“…The effectiveness of these techniques is assessed in two tasks: detecting mispronounced words (replacing, adding, removing phonemes, or pronouncing an unknown speech sound) and detecting lexical stress errors. The results presented in this study are the culmination of our recent work on speech generation in pronunciation error detection task [11,22,23], including a new S2S technique.…”

“…For example, the word 'enough' can be pronounced by native speakers in multiple ways: /ih n ah f/ or /ax n ah f/ (short 'i' or 'schwa' phoneme at the beginning). In our previous work, we solve these problems by creating a native speech pronunciation model that returns the probability of the sentence to be spoken by a native speaker [11].…”

“…Techniques based on phoneme recognition can be supplemented by a reference speech signal obtained from the speech database [37][38][39] or generated from the phonetic representation [11,40]. Xiao et al [37] use a pair of speech signals from a student and a native speaker to classify native and non-native speech.…”

“…It has been shown to help people practice and improve their pronunciation skills [6][7][8]. CAPT consists of two components: an automated pronunciation evaluation component [9][10][11] and a feedback component [12]. The automated pronunciation evaluation component is responsible for detecting pronunciation errors in spoken speech, for example, for detecting words pronounced incorrectly by the speaker.…”

“…Researcher have given most attention to studying various machine learning models such as Bayesian networks [14,15] and deep learning methods [9,10], as well as analyzing different representations of the speech signal such as prosodic features (duration, energy and pitch) [16], and cepstral/spectral features [9,13,17]. Despite significant progress in recent years, existing CAPT methods detect pronunciation errors with relatively low accuracy of 60% precision at 40%-80% recall [9][10][11]. Highlighting correctly pronounced words as pronunciation errors by the CAPT tool can demotivate students and lower the confidence in the tool.…”