Brain Prediction of Auditory Emphasis by Facial Expressions During Audiovisual Continuous Speech

Abstract

The visual cues involved in auditory speech processing are not restricted to information from lip movements but also include head or chin gestures and facial expressions such as eyebrow movements. The fact that visual gestures precede the auditory signal implicates that visual information may influence the auditory activity. As visual stimuli are very close in time to the auditory information for audiovisual syllables, the cortical response to them usually overlaps with that for the auditory stimulation; the neural dynamics underlying the visual facilitation for continuous speech therefore remain unclear. In this study, we used a three-word phrase to study continuous speech processing. We presented video clips with even (without emphasis) phrases as the frequent stimuli and with one word visually emphasized by the speaker as the non-frequent stimuli. Negativity in the resulting ERPs was detected after the start of the emphasizing articulatory movements but before the auditory stimulus, a finding that was confirmed by the statistical comparisons of the audiovisual and visual stimulation. No such negativity was present in the control visual-only condition. The propagation of this negativity was observed between the visual and fronto-temporal electrodes. Thus, in continuous speech, the visual modality evokes predictive coding for the auditory speech, which is analysed by the cerebral cortex in the context of the phrase even before the arrival of the corresponding auditory signal.

Anomalous network architecture of the resting brain in children who stutter

“…Weikum et al, (2007) reported that 4–6 month old infants can discriminate between languages (French versus English) using visual cues only (i.e., silent videos showing articulations). Electrophysiological work shows that visual cues not only enhance auditory cues (e.g., Crosse et al, 2015; see Peelle & Sommers, 2015 for a review), but that they actually precede them (Strelnikov et al, 2015). In line with this reasoning, Venezia et al (2016) propose that “exposure to visual speech during acquisition of speech production establishes the neural circuitry linking visually-perceived gestures to the speech motor system.” If visual information is used to facilitate speech acquisition, aberrant function of VN could interfere with this process.…”

Perceived Conventionality in Co-speech Gestures Involves the Fronto-Temporal Language Network

“…Indeed, the auditory cortex is modulated by dynamic and congruent visual stimuli (Zvyagintsev et al, 2009 ) and vice-versa (Wolf et al, 2014 ). During face-to-face communication recruitment of auditory cortex is not only increased by language sounds, but also cross-modally by visual input such as lip movement and facial expressions (Hertrich et al, 2007 ; Okada et al, 2013 ; Strelnikov et al, 2015 ). In the case of co-speech gestures, Hubbard and colleagues (Hubbard et al, 2009 ) showed that bilateral non-primary auditory cortex exhibited greater activity when speech was accompanied by beat gestures than when speech was presented alone.…”

Investigating Multisensory Integration in Emotion Recognition Through Bio-Inspired Computational Models

“…The Enhancement model is more biologically plausible, where the auditory part does not use prediction from the visual part but learns from the spiking patterns. This represents a multisensory learning, which helps propagate spikes from the visual group to the auditory group [67].…”