More than words: word predictability, prosody, gesture and mouth movements in natural language comprehension

Abstract

The ecology of human language is face-to-face interaction, comprising cues such as prosody, co-speech gestures and mouth movements. Yet, the multimodal context is usually stripped away in experiments as dominant paradigms focus on linguistic processing only. In two studies we presented video-clips of an actress producing naturalistic passages to participants while recording their electroencephalogram. We quantified multimodal cues (prosody, gestures, mouth movements) and measured their effect on a well-established electroencephalographic marker of processing load in comprehension (N400). We found that brain responses to words were affected by informativeness of co-occurring multimodal cues, indicating that comprehension relies on linguistic and non-linguistic cues. Moreover, they were affected by interactions between the multimodal cues, indicating that the impact of each cue dynamically changes based on the informativeness of other cues. Thus, results show that multimodal cues are integral to comprehension, hence, our theories must move beyond the limited focus on speech and linguistic processing.

The role of iconic gestures and mouth movements in face-to-face communication

“…We identified interactions between gesture, mouth informativeness, and clarity, supporting proposals in which auditory and visual cues are dynamically and flexibly weighted during communication (Skipper et al, 2009;Zhang, Frassinelli, et al, 2021b). In one of the first experimental studies looking at audiovisual (including gestures) speech, Drijvers and Özyürek (2017) demonstrated that moderately noisevocoded speech comprehension was enhanced when the two cues were present, with iconic gestures having a larger effect than mouth movements.…”

“…Thus, the face is cropped or covered in studies of gestures (e.g., Drijvers & Özyürek, 2017;Habets et al, 2010;Hirata & Kelly, 2010;Holle & Gunter, 2007;Holle et al, 2010), and the hands are not visible in studies of audiovisual speech (e.g., Ross et al, 2007;Solberg Økland et al, 2019;Tye-Murray et al, 2007). Only a handful of studies have investigated both gestures and mouth movements (Drijvers & Özyürek, 2017(Drijvers & Özyürek, , 2020Drijvers et al, 2019;Hirata & Kelly, 2010;Skipper et al, 2009;Zhang, Ding, et al, 2021a;Zhang, Frassinelli, et al, 2021b).…”

“…In another study, Zhang, Frassinelli, et al (2021b) looked at brain activity during audiovisual connected speech processing. The researchers measured changes in the N400 amplitude-a negative event-related potential associated with semantic processing (Kutas & Federmeier, 2011)-by looking at word predictability, prosodic stress, iconic gestures, beat gestures, and mouth informativeness.…”

“…The finding, albeit only marginal, of a larger mouth informativeness effect (across clarity conditions) without gestures goes beyond the work of Drijvers and Özyürek (2017), suggesting that the system weights differently the cues using the most useful at any one time. Zhang, Frassinelli, et al (2021b) showed that the N400 response evoked by words in context is modulated by the presence of different multimodal cues, such as (both iconic and beat) gestures, prosodic stress, and mouth informativeness. The researchers found that comprehension is enhanced when iconic gestures and more informative mouth movements accompany speech.…”

Aging Adults’ Motivation to Use Embodied Conversational Agents in Instrumental Activities of Daily Living: Results of Latent Profile Analysis

“…ECAs are virtual entities that can convey messages and engage in conversation in a human-like manner by using speech, language, gestures, and facial expressions [ 16 ]. While verbal channels carry symbolic/semantic interpretation of the message, the non-verbal channels (i.e., gestures, facial expressions, and prosody) serve as an orchestrator of communication and are integral to comprehension [ 17 , 18 ]. By incorporating features such as facial expressions and gestures, social links between humans and their virtual counterparts may be established.…”