Exploring the role of hand gestures in learning novel phoneme contrasts and vocabulary in a second language

Abstract

Co-speech hand gestures are a type of multimodal input that has received relatively little attention in the context of second language learning. The present study explored the role that observing and producing different types of gestures plays in learning novel speech sounds and word meanings in an L2. Naïve English-speakers were taught two components of Japanese—novel phonemic vowel length contrasts and vocabulary items comprised of those contrasts—in one of four different gesture conditions: Syllable Observe, Syllable Produce, Mora Observe, and Mora Produce. Half of the gestures conveyed intuitive information about syllable structure, and the other half, unintuitive information about Japanese mora structure. Within each Syllable and Mora condition, half of the participants only observed the gestures that accompanied speech during training, and the other half also produced the gestures that they observed along with the speech. The main finding was that participants across all four conditions had similar outcomes in two different types of auditory identification tasks and a vocabulary test. The results suggest that hand gestures may not be well suited for learning novel phonetic distinctions at the syllable level within a word, and thus, gesture-speech integration may break down at the lowest levels of language processing and learning.

Gesture and Vocabulary Learning in a Second Language

“…Our data extend and elaborate on the previous literature in several ways, however. First, the finding that high idiosyncratic gestures (i.e., noniconic gestures that can be idiosyncratically semantically associated with new words) can facilitate L2 lexical recognition as effectively as low idiosyncratic gestures (i.e., gestures that are highly iconic with established semantic meanings) adds to previous results (e.g., Kelly et al., ; Macedonia et al., , ; So et al., ). This finding is important for pedagogical reasons because large numbers of words in any language are not easily captured by iconic gestures.…”

“…Kelly, Hirata, Manansala, and Huang () also presented data suggesting that just moving the hands fails to support novel vocabulary learning, at least when the gestures are tied to low‐level phonemic properties of the novel words. Participants in this study studied Japanese vocabulary either paired or not with gestures that corresponded to either syllable structure or mora structure.…”

Metaphoric Gestures Facilitate Perception of Intonation More than Length in Auditory Judgments of Non-Native Phonemic Contrasts

“…But for more unfamiliar distinctions, such as phonemic length contrasts, metaphoric gestures may not connect smoothly with FL speech, and in fact, may actually disrupt it. Together, these contrasting FL perception findings add texture to previous work showing that metaphoric gestures do not help with learning of FL length contrasts (Hirata & Kelly, 2010;Hirata et al, 2014;Kelly & Lee, 2012;Kelly et al, 2014). We are now one step closer to identifying what aspects of FL phoneme processing are more open to the influence of gesture and what aspects are more closed.…”

Multisensory and sensorimotor interactions in speech perception

“…In a study of cued speech, i.e., specific hand-signs for different speech sounds, Bayard et al (2014) demonstrate that individuals with hearing impairment used the visual cues differently from their controls, even though both groups were experts in cued speech. Kelly et al (2014) show that when normal hearing adults learned words in a foreign language, viewing or producing hand gestures accompanying audiovisual speech did not affect the outcome. Lee and Noppeney (2014) show that musicians had a narrower audiovisual temporal integration window for music, and to a smaller extent also for speech, implying that the effect transfers from the practiced music stimuli also to other stimulus types.…”

“…In a study of cued speech, i.e., specific hand-signs for different speech sounds, Bayard et al (2014) demonstrate that individuals with hearing impairment used the visual cues differently from their controls, even though both groups were experts in cued speech. Kelly et al (2014) Tiippana et al …”