Evaluation of synthetic and natural Mandarin visual speech: Initial consonants, single vowels, and syllables

Chen, Trevor H.; Massaro, Dominic W.

doi:10.1016/j.specom.2011.03.009

Cited by 12 publications

(3 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Among these visual cues, the most speech-specific cue is viseme. Chen and Massaro ( 2011 ) defined visemes as “units/categories of visual speech movements that were perceptually distinctive among the units/categories but much less so within each unit/category” (p. 956). Viseme is a well-established concept in audiovisual speech perception (Bernstein, 2012 ).…”

Section: Introductionmentioning

confidence: 99%

Incongruent visual cues affect the perception of Mandarin vowel but not tone

Hong

Wang²,

Zeng³

2023

Front. Psychol.

View full text Add to dashboard Cite

Over the recent few decades, a large number of audiovisual speech studies have been focusing on the visual cues of consonants and vowels but neglecting those relating to lexical tones. In this study, we investigate whether incongruent audiovisual information interfered with the perception of lexical tones. We found that, for both Chinese and English speakers, incongruence between auditory and visemic mouth shape (i.e., visual form information) significantly interfered with reaction time and reduced the identification accuracy of vowels. However, incongruent lip movements (i.e., visual timing information) did not interfere with the perception of auditory lexical tone. We conclude that, in contrast to vowel perception, auditory tone perception seems relatively impervious to visual congruence cues, at least under these restricted laboratory conditions. The salience of visual form and timing information is discussed based on this finding.

show abstract

Section: Introductionmentioning

confidence: 99%

Incongruent visual cues affect the perception of Mandarin vowel but not tone

Hong

Wang²,

Zeng³

2023

Front. Psychol.

View full text Add to dashboard Cite

show abstract

“…Another key issue of using a virtual 3‐D talking head is whether such an audiovisual CAPT system can enhance the learners' pronunciation skills in reality. Investigations into Mandarin‐specific phonetic characteristics and applying Mandarin visual speech in audiovisual training present potentially unique and interesting contributions (Chen & Massaro, 2011). Recent developments in synthetic Mandarin visual speech have been shown to provide an effective way to facilitate Mandarin pronunciation through demonstration of internal and external articulator animations by a talking head (Liu et al, 2013; Peng et al, 2018).…”

Section: Introductionmentioning

confidence: 99%

Learning challenging L2 sounds via computer‐assisted training: Audiovisual training with an airflow model

Chen

Xia

Yan

et al. 2022

Computer Assisted Learning

View full text Add to dashboard Cite

Background Teaching Mandarin as a second language (L2) has become an important profession and an important research area. The acquisition of unaspirated and aspirated consonants in Mandarin has been reported to be rather challenging for L2 learners. Objectives In the current study, a 3‐D airflow model was integrated into the virtual talking head for audiovisual pronunciation training in these Mandarin consonants. Methods Using the eye‐tracking technique, Experiment 1 investigated L2 learners' general acceptance and gauged attention distribution online while learning with the system. To further compare training efficacy, learning outcomes were evaluated by randomly dividing Japanese learners of Mandarin into different training groups using the 3‐D tutor with and without an airflow model, respectively, in Experiment 2. Results and Conclusions Eye‐tracking results showed that the talking head was well accepted, and was successful in directing L2 learners' attention to the visual modality of the airflow model marking the contrast in aspiration. Furthermore, the additional visual cue of the airflow model helped enhance their production of the aspirated Mandarin stops. Moreover, this computer‐assisted training approach was shown to be robust as the advantage of training with an airflow model can be generalized to novel syllables with a change of tones or rimes.

show abstract

“…Importantly, investigations on Mandarin visual speech potentially make unique contributions [17], since some kinematic characteristics and acoustic features of Mandarin phonemes are different compared with other languages. Besides the supra-segmental lexical tones, Mandarin phonology system has a variety of voiceless stops and affricates with similar places of articulation, which were difficult for non-Mandarin second language (L2) learners to acquire [18]- [20].…”

mentioning

confidence: 99%

Investigations on Mandarin Aspiratory Animations Using an Airflow Model

Chen

Wang

Chen

et al. 2017

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

Various 3-D talking heads have been developed lately for language learning, with both external and internal articulatory movements being visualized to guide learning. Mandarin pronunciation animation is challenging due to its confusable stops and affricates with similar places of articulation. Until now, less attention has been paid to the bio-signal information of aspiratory airflow, which is essential in distinguishing Mandarin consonants. This study fills a research gap by presenting the quantitative analyses of airflow, and then designing an airflow model for a 3-D pronunciation system. The airflow information was collected by Phonatory Aerodynamic System (PAS), so that confusable consonants in Mandarin could be discerned by means of airflow rate, peak airflow rate, airflow duration, and peak time. Based on the airflow parameters, an airflow model using the physical equation of fluid flow was proposed and solved, which was then combined and synchronized with the existing 3-D articulatory model. Therefore, the new multimodal system was implemented to synchronously exhibit the airflow motions and articulatory movements of uttering Mandarin syllables. Both an audio-visual perception test and a pronunciation training study were conducted to assess the effectiveness of our system. Perceptual results indicated that identification accuracy was improved for both native and non-native groups with the help of airflow motions, while native perceivers exhibited higher accuracy due to long-term language experience. Moreover, our system helped Japanese learners of Mandarin enhance their production skills of Mandarin aspirated consonants, reflected by higher gain values of voice onset time (VOT) after training.

show abstract

Evaluation of synthetic and natural Mandarin visual speech: Initial consonants, single vowels, and syllables

Cited by 12 publications

References 21 publications

Incongruent visual cues affect the perception of Mandarin vowel but not tone

Incongruent visual cues affect the perception of Mandarin vowel but not tone

Learning challenging L2 sounds via computer‐assisted training: Audiovisual training with an airflow model

Investigations on Mandarin Aspiratory Animations Using an Airflow Model

Contact Info

Product

Resources

About