Perceived Conventionality in Co-speech Gestures Involves the Fronto-Temporal Language Network

Abstract

Face-to-face communication is multimodal; it encompasses spoken words, facial expressions, gaze, and co-speech gestures. In contrast to linguistic symbols (e.g., spoken words or signs in sign language) relying on mostly explicit conventions, gestures vary in their degree of conventionality. Bodily signs may have a general accepted or conventionalized meaning (e.g., a head shake) or less so (e.g., self-grooming). We hypothesized that subjective perception of conventionality in co-speech gestures relies on the classical language network, i.e., the left hemispheric inferior frontal gyrus (IFG, Broca's area) and the posterior superior temporal gyrus (pSTG, Wernicke's area) and studied 36 subjects watching video-recorded story retellings during a behavioral and an functional magnetic resonance imaging (fMRI) experiment. It is well documented that neural correlates of such naturalistic videos emerge as intersubject covariance (ISC) in fMRI even without involving a stimulus (model-free analysis). The subjects attended either to perceived conventionality or to a control condition (any hand movements or gesture-speech relations). Such tasks modulate ISC in contributing neural structures and thus we studied ISC changes to task demands in language networks. Indeed, the conventionality task significantly increased covariance of the button press time series and neuronal synchronization in the left IFG over the comparison with other tasks. In the left IFG, synchronous activity was observed during the conventionality task only. In contrast, the left pSTG exhibited correlated activation patterns during all conditions with an increase in the conventionality task at the trend level only. Conceivably, the left IFG can be considered a core region for the processing of perceived conventionality in co-speech gestures similar to spoken language. In general, the interpretation of conventionalized signs may rely on neural mechanisms that engage during language comprehension.

Atypical MEG inter-subject correlation during listening to continuous natural speech in dyslexia

“…In terms of phoneme processing, the left middle temporal cortex would be expected to play a major role, as it is an integral part of speech and word processing (Hickok and Poeppel, 2007). In fMRI studies, the peak location for differences between our groups found for the beta band has been frequently associated with activations during listening to speech in various ways (Narain et al, 2003;Oechslin et al, 2010;Straube et al, 2013b;Nagels et al, 2013;Evans et al, 2016, Wolf et al, 2017.…”

Why We Should Study Multimodal Language

“…Rather all kinds of context, including multimodal cues, are processed simultaneously and immediately (Hagoort and van Berkum, 2007 ). Numerous studies have provided evidence for similar processing of gesture and speech in terms of semantic and temporal integration (Özyürek et al, 2007 ; Hubbard et al, 2009 ; Straube et al, 2009 ; Habets et al, 2011 ; Dick et al, 2014 ; Yang et al, 2015 ; Peeters et al, 2017 ), as well as in terms of perceiving conventionalized meaning (Andric et al, 2013 ; Wolf et al, 2017 ). In addition, there is evidence that prosodic information from visual and vocal channels is treated similarly by the brain, with gestural beats functioning as visual prosody complementary to speech prosody (Biau et al, 2016 ).…”

The Role of Iconic Gestures in Speech Comprehension: An Overview of Various Methodologies

“…To date, studies on gesture-speech integration have employed diverse methodologies, whether in terms of the definition for iconic gestures used, the task or even instructions given to participants. And as suggested by Wolf et al (2017) , the interpretation of verbal and gestural information can be modulated according to the task and/or instruction given to the participants. Our aim is, therefore, to put into perspective the data found in the field of iconic gesture-speech integration by specifically highlighting the methodological variations.…”

“…Our aim is, therefore, to put into perspective the data found in the field of iconic gesture-speech integration by specifically highlighting the methodological variations. Indeed, the diversity of results yielded in this field could be explained by (1) non-identical testing methods ( Wolf et al, 2017 ) (2) overlooking the specificities of iconic gestures or (3) a non-integrative interpretation of results.…”