Effects of hand gesture and lip movements on auditory learning of second language speech sounds

Abstract

Previous research found that auditory training helps native English speakers to perceive phonemic vowel length distinction in Japanese, but that their performance has never reached native levels (Hirata et al., 2007). Given that multimodal information, such as hand gesture and lip movements, influences semantic aspects of language processing and development (Kelly et al., 2002), we examined whether this multimodal information helps to improve native English speakers' ability to perceive Japanese vowel length distinction. Forty-five native English speakers participated in one of three types of training: (1) audio alone; (2) audio with hand gestures; and (3) audio with lip movements and hand gestures. Before and after training, participants were given phoneme perception tests that measured their ability to distinguish between short and long vowels in Japanese, e.g., /kato/ versus /kato:/. Our original prediction was that more modalities in training would result in greater learning. Although all three groups improved from pre- to post-test, there were no significant differences among the three training groups. Unlike the original prediction, hand gestures and lip movements did not seem to augment learning of difficult sound distinctions. We will discuss possible benefits and limitations of using multimodal information in second language speech learning.

Visual Context Enhanced: The Joint Contribution of Iconic Gestures and Visible Speech to Degraded Speech Comprehension

“…In earlier work on the contribution of visible speech and hand gestures to learning nonnative speech sounds, Kelly, Hirata, et al (2008) argued that lip and mouth movements help in auditory encoding of speech, whereas hand gestures can only help to understand the meaning of words in the speech stream when the auditory signal is correctly encoded. On the basis of their results, Kelly, Hirata, et al (2008) argued that the benefits of multimodal input target different stages of linguistic processing.…”

“…On the basis of their results, Kelly, Hirata, et al (2008) argued that the benefits of multimodal input target different stages of linguistic processing. Here, mouth movements seem to aid during phonological stages, whereas hand gestures aid during semantic stages, which, according to the authors, fits with McNeill's (1992) interpretation of speech and gesture forming an integrated system during language comprehension.…”

“…Indeed, visible speech possibly plays a significant role during auditory encoding of speech, but according to our current results, iconic gestures not only benefit comprehension when auditory information can be correctly encoded and understood, but also benefit comprehension under adverse listening conditions (cf. Kelly, Hirata, et al, 2008). Even in 2-band noise-vocoding, when auditory cues are no longer reliable, and correct encoding of the auditory input is difficult, gestures significantly enhance comprehension.…”

“…However, it remains unknown how these two articulators interact when both are able to enhance language comprehension in adverse listening conditions. Two other studies, by Kelly, Hirata, et al (2008) and Hirata and Kelly (2010), examined the effects of lip movements and iconic gestures on auditory learning of second-language speech sounds (i.e., prosody and segmental phonology of Japanese). They hypothesized that having both modalities present would benefit learning the most, but they found that only lip movements resulted in greater learning.…”

The Impact of Videos Presenting Speakers’ Gestures and Facial Clues on Iranian EFL Learners’ Listening Comprehension

“…There are a number of researches devoted to this field of study which demonstrate the significant role of non-verbal communication in general and gestures and facial clues in particular in listening comprehension. Researchers such as, Hirata and Kelly (2010), Kruger (2009), Kelly, Hirata, Simester, Burch, Cullings and Demakakos (2008), Kusanagi (2005), Sueyoshi and Hardison (2005), and Tae (1993) have done valuable studies in the area. Kruger (2009) illustrates the noticeable role that non verbal behavior plays in communication.…”