Behavioural and neurophysiological evidence of semantic interaction between iconic gestures and words

Abstract

We report two experiments that provide converging behavioural and neurophysiological evidence on the relationship between the meaning of iconic gestures and words. Experiment 1 exploited a semantic priming paradigm and revealed interference between gestures and words when they were not related in meaning, but no facilitation when they were. This result was confirmed in Experiment 2, where ERPs were recorded during silent word reading with the same paradigm. The analysis showed a negative deflection peaking near 400 ms (N400) and, in the left hemisphere, greater negative values for verbs than for nouns. Differently from the classical distribution obtained with verbal stimuli, we found an N400 that spread more over central-anterior areas of the scalp, suggesting that the meaning systems of gesture and language do not overlap completely. These results are consistent with the view that the meaning systems for gesture and speech are tightly integrated.

The Role of Synchrony and Ambiguity in Speech–Gesture Integration during Comprehension

“…This large latency window has been found in previous gesture-speech integration studies with regard to the N400 (Bernardis et al, 2008;Wu & Coulson, 2005). Match and mismatch for this condition are similar on the N1-P2 complex, but they start to diverge at the N400 component, with mismatch stimuli being more negative in amplitude compared with match stimuli (see also Figure 2 for the mismatch-match difference wave).…”

“…Gestures can also contribute to building a context into which subsequent words can be semantically integrated. Words that did not semantically match the preceding gesture elicited a larger N400 than those that semantically matched (Bernardis, Salillas, & Caramelli, 2008;Kelly, Kravitz, & Hopkins, 2004). In addition, words that did not semantically match the preceding gesture-speech combination elicited a larger N400 than those that matched (Holle & Gunter, 2007).…”

“…In Kelly et al (2004), the gesture onset preceded the speech onset by 800 msec. Similarly, in Bernardis et al (2008) and Coulson (2005, 2007a), a gesture preceded a visual presentation of a word by a 1000-msec ISI. These three studies found an N400 effect because of the semantic congruity of gesture and linguistic stimuli, unlike the SOA 360 condition in the present study.…”

“…Previous studies have shown that speech and gesture can be integrated into a stable (preexisting) and unambiguous context, similarly to how a visually presented word can be integrated into a preceding sentential context (Kutas & Hillyard, 1980). For example, they have shown that a word is integrated into a preceding gesture (Bernardis et al, 2008;Wu & Coulson, 2005;Kelly et al, 2004), that a gesture is integrated into the context created by a preceding sentence (Özyurek et al, 2007), and that a picture probe or a word was integrated with a preceding gesture-speech combination (Holle & Gunter, 2007;Wu & Coulson, 2007b). When previously the integration of concurrent speech and gesture was demonstrated (Kelly, Ward, Creigh, & Bartolotti, 2007;Kelly et al, 2004), a small number of predictable gestures and words were repeated a number of times, and thus the gestures were not as ambiguous as they usually are.…”

Integrating Speech and Iconic Gestures in a Stroop-like Task: Evidence for Automatic Processing

“…Many of the initial studies used ERPs to investigate the time course of gesture-speech integration (Bernardis, Salillas, & Caramelli, 2008;Holle & Gunter, 2007;Özyürek, Willems, Kita, & Hagoort, 2007;Wu & Coulson, 2007a, 2007bKelly, Kravitz, & Hopkins, 2004). For example, Wu and Coulson (2007b) presented gesture-speech utterances followed by pictures that were related either to gesture and speech or to just the speech alone.…”