Speech-gesture mismatches: Evidence for one underlying representation of linguistic and nonlinguistic information

Abstract

Adults and children spontaneously produce gestures while they speak, and such gestures appear to support and expand on the information communicated by the verbal channel. Little research, however, has been carried out to examine the role played by gesture in the listener's representation of accumulating information. Do listeners attend to the gestures that accompany narrative speech? In what kinds of relationships between gesture and speech do listeners attend to the gestural channel? If listeners do attend to information received in gesture, how is this information represented-is it 'tagged' as originating in the gestural channel? In this article research is described that addresses these questions. Results show that listeners do attend to information conveyed in gesture, when that information supplements or even contradicts the information conveyed by speech. And information received via gesture is available for retelling in speech. These results are taken to demonstrate that gesture is not taken by the listener to be epiphenomenal to the act of speaking, or a simple manual translation of speech. But they also suggest that the information conveyed in a discourse may be represented in a manner that is neither gesture nor language, although accessible to both channels.Pantomime without discourse will leave you nearly tranquil, discourse without gestures will wring tears from you.

Development of multimodal discourse comprehension: cohesive use of space by gestures

“…Second, we provide evidence that adults integrate information from spoken discourse and cohesive gestures. This finding is in line with previous studies conducted in English, which showed that adult listeners take into account information conveyed by a speaker's gestures that are anaphorically used (Cassell et al, 1999;Goodrich & Hudson-Kam, 2012;McNeill et al, 1994).…”

“…Though no previous studies investigated children's comprehension of cohesive gestures, studies have shown that adult listener-viewers take up information from the cohesive use of space in gesture. McNeill and his colleagues (Cassell, McNeill, & McCullough, 1999;McNeill, Cassell, & McCullough, 1994) presented a video-recorded narrative to adult participants, who then re-told the story to a listener. In the stimulus narrative, the narrator set up two referents in the frontal space of the speaker with deictic gestures, and then linguistically referred back to one of the referents, but pointed to the wrong space (the space for the other referent) at the same time.…”

Consistency of personality in interactive characters: verbal cues, non-verbal cues, and user characteristics

“…In addition, studies that looked at how people detect deception have found that people turn to non-verbal cues to see if they are inconsistent with the verbal ones (Ekman & Friesen, 1974;Ekman, et al, 1980). Finally, researchers have shown that although people are not necessarily aware of mismatches between verbal and gestural cues, they integrate both of these sets of cues into their understanding of what was said (Cassell et al, 1998). These results taken together suggest that discrepancy among cues is something that will be noticed, and that may be a critical problem in other people.…”

“…People also convey important conversational information through their gestures. Most notable in this regard is Justine Cassell's research program, which investigates the importance and automation of these sorts of communicative gestures in interactive characters (Cassell, McNeill & McCullough, 1998;Cassell & Thorisson, 2000;Cassell & Vilhjalmsson, 1999). …”

Predicting Head Pose from Speech with a Conditional Variational Autoencoder

“…Research suggests that speech and gesture stem from the same internal process and share the same semantic meaning [1,2].…”