The Role of Iconic Gestures in Speech Disambiguation: ERP Evidence

Abstract

The present series of experiments explored the extent to which iconic gestures convey information not found in speech. Electroencephalogram (EEG) was recorded as participants watched videos of a person gesturing and speaking simultaneously. The experimental sentences contained an unbalanced homonym in the initial part of the sentence (e.g., She controlled the ball ...) and were disambiguated at a target word in the subsequent clause (which during the game ... vs. which during the dance ...). Coincident with the initial part of the sentence, the speaker produced an iconic gesture which supported either the dominant or the subordinate meaning. Event-related potentials were time-locked to the onset of the target word. In Experiment 1, participants were explicitly asked to judge the congruency between the initial homonym-gesture combination and the subsequent target word. The N400 at target words was found to be smaller after a congruent gesture and larger after an incongruent gesture, suggesting that listeners can use gestural information to disambiguate speech. Experiment 2 replicated the results using a less explicit task, indicating that the disambiguating effect of gesture is somewhat task-independent. Unrelated grooming movements were added to the paradigm in Experiment 3. The N400 at subordinate targets was found to be smaller after subordinate gestures and larger after dominant gestures as well as grooming, indicating that an iconic gesture can facilitate the processing of a lesser frequent word meaning. The N400 at dominant targets no longer varied as a function of the preceding gesture in Experiment 3, suggesting that the addition of meaningless movements weakened the impact of gesture. Thus, the integration of gesture and speech in comprehension does not appear to be an obligatory process but is modulated by situational factors such as the amount of observed meaningful hand movements.

Neural correlates of the processing of co-speech gestures

“…When the hand movements offered no helpful cue for the interpretation of the sentence (i.e., in the grooming condition), word meaning frequency had a significant (albeit small) influence on target word selection. These results are in line with our previous findings in showing that listeners use the information provided by iconic gestures to disambiguate speech (Holle and Gunter, 2007). In the absence of a cue for meaning selection, word meaning frequency influences which meaning of the ambiguous sentence is selected (Holle and Gunter, 2007, Exp. 3).…”

“…In light of the rather low amount of dominant target word selections after grooming videos (just above chance level), it is a possibility that word meaning frequency was not effectively varied in this study. Note, however, that in a number of previous experiments, the same set of homonyms elicited strong effects of word meaning frequency (Gunter et al, 2003;Holle and Gunter, 2007).…”

“…In a series of previously conducted ERP experiments, we have shown that such additional gesture information can modulate how the two word meanings of lexically ambiguous words (e.g. ball) are processed in a sentence context (Holle and Gunter, 2007). Thus, iconic gestures interact with speech during online language comprehension.…”

“…Previous research has shown that listeners make use of additional gesture information in comprehending language (Alibali et al, 1997;Shovelton, 1999, 2002). Furthermore, progress has been made in specifying the temporal characteristics of gesture-speech interaction in comprehension (Holle and Gunter, 2007;Kelly et al, 2004;Özyürek et al, 2007;Wu and Coulson, 2007). The present study investigates the neural systems involved in the interaction of gesture and speech in comprehension.…”

Event-related brain potentials during the monitoring of speech errors

“…pictures, line drawings, or gestures) into a context have reported a more frontal scalp distribution for the N400 effects (e.g. Federmeier, and Kutas, 2001;Ganis, Kutas, and Sereno, 1996;Holle, and Gunter, 2007;Willems et al, 2007). Here, it is suggested that in Experiment 2 the additional attempt to predict upcoming words involves (parts of) the language production system (Pickering, and Garrod, 2007).…”