Development of an articulatory visual-speech synthesizer to support language learning

Abstract

-This paper presents a two-dimensional (2D) visualspeech synthesizer to support language learning. A visual-speech synthesizer animates the human articulators in synchronization with speech signals, e.g., output from a text-to-speech synthesizer. A visual-speech animation can offer a concrete illustration to the language learners on how to move and where to place the articulators when pronouncing a phoneme. We adopt a 2D vector-based viseme models and compiled a collection of visemes to cover the articulation of all English phonemes (42 visemes for the 44 English phonemes). Morphing between properly selected vector-based articulation images achieves articulatory animations. In this way, we have developed an articulatory visual speech synthesizer that can accept free-text input and synthesize articulatory dynamics in real-time. Evaluation involving 32 subjects based on "lip-reading" shows that they can identify the appropriate word(s) based on articulation animation alone nearly ~80% of the time

Audiovisual synthesis of exaggerated speech for corrective feedback in computer-assisted pronunciation training

“…Our exaggerated feedback is implemented on the previous visual-speech synthesizer reported in [9][10], which visualized pronunciation movement from midsagittal and front views of the vocal tract. It focuses on providing corrective feedback and can offer a reliable visualization for coarticulation.…”

“…In our visual-speech synthesis system, each phoneme is assigned to two visemes as the key frames for animation generation [9][10]. Each viseme can be assumed as the representation of a key articulatory action.…”

“…The morphing results between the k th viseme V k and the (k + 1) th viseme V k+1 are generated according to (9), where w(t) is the blending weight at time t. To emphasize one key action, we slow down the rate of change in blending weights around the corresponding viseme in focus. To be specific, let the two visemes indexed by n and (n + 1) belong to one phoneme and we will emphasize its first viseme V n by non-linear morphing.…”

Effective computer‐assisted pronunciation training based on phone‐sensitive word recommendation

“…Learning approaches to pronunciation can be roughly divided into two types: phonics training and whole-word training. 13 The phonics training approach emphasizes phoneme training and aims to correct phoneme-level errors. 14 In contrast, whole-word training emphasizes meaning to encourage memorization by students.…”

Allophonic variations in visual speech synthesis for corrective feedback in CAPT

“…WASAY generates synchronized animations of the speech articulators in the midsagittal and the front views. The initial implementation of WASAY [6] uses context-independent visemes, and blending such visemes does not offer a reliable visualization for coarticulation.…”