Early Binding of Gaze, Gesture, and Emotion: Neural Time Course and Correlates

Abstract

Communicative intentions are transmitted by many perceptual cues, including gaze direction, body gesture, and facial expressions. However, little is known about how these visual social cues are integrated over time in the brain and, notably, whether this binding occurs in the emotional or the motor system. By coupling magnetic resonance and electroencephalography imaging in humans, we were able to show that, 200 ms after stimulus onset, the premotor cortex integrated gaze, gesture, and emotion displayed by a congener. At earlier stages, emotional content was processed independently in the amygdala (170 ms), whereas directional cues (gaze direction with pointing gesture) were combined at ϳ190 ms in the parietal and supplementary motor cortices. These results demonstrate that the early binding of visual social signals displayed by an agent engaged the dorsal pathway and the premotor cortex, possibly to facilitate the preparation of an adaptive response to another person's immediate intention.

Selective attention effects on early integration of social signals: Same timing, modulated neural sources

“…Of interest, once merged into a single percept, co-emitted social signals can take on new significance and certain combinations, notably those indicating the presence of threat, become more relevant than others. Typically, angry expressions are perceived as more threatening when associated with a direct gaze than with an averted gaze (Sander et al 2007; N' Diaye et al 2009;Sato et al 2010;Conty et al 2012). Despite the clear relevance of understanding the spatiotemporal characteristics of the mechanisms underlying the combination of social cues, the question of whether, when and how, directing one's attention towards a specific social signal impacts on its neural integration with other co-emitted social cues remains.…”

“…One possible explanation is that the neural sources underlying social cues integration may be task demand-dependent (Graham and Labar 2012;Dumas et al 2013). While most of the functional Magnetic Resonance Imagery (fMRI) experiments revealed that the amygdala integrated emotion and gaze when participants were required to attend to the emotional content of the faces or their gender (Sato et al 2004(Sato et al , 2010Sander et al 2007; N' Diaye et al 2009;Adams et al 2012), the premotor cortex was involved in gaze and expression combination when participants were requested to attend to gaze direction (Conty et al 2012).…”

“…Electroencephalography (EEG) results also suggest that the temporal marker of such integration is susceptible to task demand influences: it emerges around 170 ms after stimulus onset when participants attend emotional expressions (Akechi et al 2010), at around 200 ms when participants attend gaze direction (Conty et al 2012), at 240 ms during passive viewing (Rigato et al 2010), and later during an oddball task concerning gaze direction (Klucharev and Sams 2004). Finally, other studies in the literature have reported that when observers orient their attention to only one feature at a time as requested by the task (i.e.…”

Linking language to the visual world: Neural correlates of comprehending verbal reference to objects through pointing and visual cues

“…Following the direction of someone's pointing finger elicits bilateral posterior superior temporal sulcus (pSTS) activation (Materna et al, 2008). Integrating someone's pointing gestures with their gaze direction recruits parietal and supplementary motor cortices in the right hemisphere (Conty et al, 2012). Together, these findings suggest an extensive right-hemisphere dominant network that is activated when one perceives a manual pointing gesture that shifts one's attention.…”

“…Previous neuroimaging studies have looked at the neural correlates of observing pointing gestures outside a referential speech context and at their integration with cues such as the gesturer's gaze direction (e.g., Brunetti et al, 2014;Conty et al, 2012;Gredebäck et al, 2010;Materna et al, 2008;Redcay et al, 2015;Sato et al, 2009). Perceiving a pointing hand compared to perceiving a non-directional closed hand elicits enhanced activation in a set of mainly right-hemisphere regions, including right inferior frontal gyrus (IFG), right angular gyrus, right parietal lobule, right thalamus, and bilateral lingual gyri (Sato et al, 2009).…”