The processing of speech, gesture, and action during language comprehension

Abstract

Hand gestures and speech form a single integrated system of meaning during language comprehension, but is gesture processed with speech in a unique fashion? We had subjects watch multimodal videos that presented auditory (words) and visual (gestures and actions on objects) information. Half of the subjects related the audio information to a written prime presented before the video, and the other half related the visual information to the written prime. For half of the multimodal video stimuli, the audio and visual information contents were congruent, and for the other half, they were incongruent. For all subjects, stimuli in which the gestures and actions were incongruent with the speech produced more errors and longer response times than did stimuli that were congruent, but this effect was less prominent for speech-action stimuli than for speech-gesture stimuli. However, subjects focusing on visual targets were more accurate when processing actions than gestures. These results suggest that although actions may be easier to process than gestures, gestures may be more tightly tied to the processing of accompanying speech.

Processing language in face-to-face conversation: Questions with gestures get faster responses

“…This information is taken up by recipients (e.g., Holler, Shovelton, & Beattie, 2009;S. D. Kelly, Barr, Church, & Lynch, 1999) and readily integrated with the information from the spoken channel (e.g., Kelly, Healey, Özyürek, & Holler, 2015;Kelly, Kravitz, & Hopkins, 2004;Willems, Özyürek, & Hagoort, 2007). Importantly, receiving gestural information in addition to speech appears to facilitate information processing in experimental settings as evidenced by faster reaction times to speech-plus-gesture compared to speech-only stimuli (Holle, Gunter, Rüschemeyer, Hennenlotter, & Iacoboni, 2008;Kelly, Özyürek, & Maris, 2010, Note 2;Nagels, Kircher, Steines, & Straube, 2015;Wu & Coulson, 2015).…”

Gesture as representational action: A paper about function

“…For example, although adults can seamlessly and easily integrate information conveyed in speech with gesture, they often fail to integrate that information with instrumental action. For example, adults can easily ignore actions that are incongruent with the speech with which they are produced, but they have difficulty ignoring gestures that are incongruent with the speech they accompany, suggesting a difference in the relative strength of speech-gesture integration vs. speech-action integration (Kelly, Healy, Özyürek, & Holler, 2014). Thus, gesture has a different relationship to speech than instrumental action does and, in turn, has a different effect on listeners than instrumental action.…”

What makes a movement a gesture?

“…In other words, gesture does not depict a change in the world (e.g., opening a jar by twisting it), or display patterns of movement (e.g., performing steps in a dance), but instead represents movement that could change the world (e.g., a gesture showing how the jar could be twisted open) or represents movement that stands on its own (e.g., a gesture showing how the dance should be performed). Importantly, observers respond differently to gesture than to other types of movements (Dick, Goldin-Meadow, Hasson, Skipper, & Small, 2009; Kelly, Healy, Ozyurek, & Holler, 2014), and these differences in response can have an impact on thinking and learning (Novack, Congdon, Hemani-Lopez, & Goldin-Meadow, 2014; Trofatter, Kontra, Beilock, & Goldin-Meadow, 2014). Our goal is to determine the conditions under which a distinction between gesture and other types of movement is made, and to explore how those conditions contribute to a top-down categorization of movement as gesture.…”