Beat that Word: How Listeners Integrate Beat Gesture and Focus in Multimodal Speech Discourse

Abstract

Abstract■ Communication is facilitated when listeners allocate their attention to important information (focus) in the message, a process called "information structure." Linguistic cues like the preceding context and pitch accent help listeners to identify focused information. In multimodal communication, relevant information can be emphasized by nonverbal cues like beat gestures, which represent rhythmic nonmeaningful hand movements. Recent studies have found that linguistic and nonverbal attention cues are integrated independently in single sentences. However, it is possible that these two cues interact when information is embedded in context, because context allows listeners to predict what information is important. In an ERP study, we tested this hypothesis and asked listeners to view videos capturing a dialogue. In the critical sentence, focused and nonfocused words were accompanied by beat gestures, grooming hand movements, or no gestures. ERP results showed that focused words are processed more attentively than nonfocused words as reflected in an N1 and P300 component. Hand movements also captured attention and elicited a P300 component. Importantly, beat gesture and focus interacted in a late time window of 600-900 msec relative to target word onset, giving rise to a late positivity when nonfocused words were accompanied by beat gestures. Our results show that listeners integrate beat gesture with the focus of the message and that integration costs arise when beat gesture falls on nonfocused information. This suggests that beat gestures fulfill a unique focusing function in multimodal discourse processing and that they have to be integrated with the information structure of the message. ■

Encouraging children to produce rhythmic beat gestures leads to better narrative discourse performances

“…Our findings add to the existing literature a clear indication of the usefulness of not only observing [as in 1] but also producing beat gestures in a brief narrative discourse training. This constitutes further evidence that beat gestures serve as visual highlighters of linguistic functions that are associated with focus marking, rhythmic marking, and discourse structure marking [29,30], confirming electrophysiological studies showing that beats are connected to language-related areas of the brain (e.g., [40]). …”

“…We hypothesize that it will, given the evidence that beat gestures play an important 9th International Conference on Speech Prosody 2018 13-16 June 2018, Poznań, Poland role as highlighters of linguistic functions [29,30] such as focus marking, rhythmic marking, and discourse structure marking. Thus, we suggest that beats are pragmatically meaningful gestures [11,31,32] that can help build up children's narrative performance.…”

Beat Gestures and Syntactic Parsing: An ERP Study

“…In the present study, we might have affected the phrase final position intonations, relevant to anchor beats to acoustic envelope modulations. Recently, Dimitrova et al (2016) showed that when a beat accompanied a nonaccented target word, it resulted in a greater late anterior positivity relative to a beat accompanying the equivalent focused word. This result actually suggested an increased cost of multimodal speech processing when nonverbal information targeted verbal information that was not auditorily emphasized.…”

Semi-Automated 8 Collaborative Online Training Module for Improving Communication Skills

“…Vocalized pauses or fillers such as "ah" or "um" are distractions that may express nervousness or lack of sincerity of the speaker [9]. The meaning of behavior could be strengthened or contradicted when they are combined with multimodality [10,11]. For example, expressions of amusement can be detected by observing a speaker's facial expression but the effect can be bolstered by the speaker's voice [11].…”