Large Scale Functional Brain Networks Underlying Temporal Integration of Audio-Visual Speech Perception: An EEG Study

Abstract

Observable lip movements of the speaker influence perception of auditory speech. A classical example of this influence is reported by listeners who perceive an illusory (cross-modal) speech sound (McGurk-effect) when presented with incongruent audio-visual (AV) speech stimuli. Recent neuroimaging studies of AV speech perception accentuate the role of frontal, parietal, and the integrative brain sites in the vicinity of the superior temporal sulcus (STS) for multisensory speech perception. However, if and how does the network across the whole brain participates during multisensory perception processing remains an open question. We posit that a large-scale functional connectivity among the neural population situated in distributed brain sites may provide valuable insights involved in processing and fusing of AV speech. Varying the psychophysical parameters in tandem with electroencephalogram (EEG) recordings, we exploited the trial-by-trial perceptual variability of incongruent audio-visual (AV) speech stimuli to identify the characteristics of the large-scale cortical network that facilitates multisensory perception during synchronous and asynchronous AV speech. We evaluated the spectral landscape of EEG signals during multisensory speech perception at varying AV lags. Functional connectivity dynamics for all sensor pairs was computed using the time-frequency global coherence, the vector sum of pairwise coherence changes over time. During synchronous AV speech, we observed enhanced global gamma-band coherence and decreased alpha and beta-band coherence underlying cross-modal (illusory) perception compared to unisensory perception around a temporal window of 300–600 ms following onset of stimuli. During asynchronous speech stimuli, a global broadband coherence was observed during cross-modal perception at earlier times along with pre-stimulus decreases of lower frequency power, e.g., alpha rhythms for positive AV lags and theta rhythms for negative AV lags. Thus, our study indicates that the temporal integration underlying multisensory speech perception requires to be understood in the framework of large-scale functional brain network mechanisms in addition to the established cortical loci of multisensory speech perception.

Neurodynamic explanation of inter-individual and inter-trial variability in cross-modal perception

“…Alpha and gamma band coherences are observed in processing of multisensory stimulus (Hummel & Gerloff, 2005; Kanayama et al, 2007; Doesburg et al 2008; Kayser et al, 2008; Maier et al, 2008; Kayser & Logothetis, 2009; Kumar et al, 2016; also present results, Figure 2 ). Interestingly, high gamma coherence is seen when the nature of multisensory stimulus is complex (asynchronous, incongruent) (Doesburg et al, 2008; Kumar et al, 2016 and present results, Figure 2 ) which in some instances lead to illusory perception (Kanayama et al, 2007; Kumar et al, 2016; and present results, Figure 1 ). Gamma coherence is also observed in the communication involving higher order multisensory areas (Maier et al, 2008; and present results, Figure 4 ).…”

“…Hence, it is important to identify which of these are meaningful to the ongoing task and the subtle differences that vary with the context in which the task evolves. In an earlier study Kumar et al (Kumar et al, 2016) have showed that global coherogram captures the difference in processing of crossmodal (illusory / ta /) and unimodal (non-illusory / pa /) perception in frequent perceivers from a subset of data that we present in this manuscript. While the detailed pattern of coherogram differences between / ta / and / pa / trials in perceivers and rare perceivers are slightly different, there was an enormous similarity in trend of coherence differences in distinct spectro-temporal locations that was conspicuous.…”

“…Neurophysiological studies highlight the pre-stimulus activity in STS and its functional connectedness to front-parietal regions as a neuromarker of illusory perception within a group of individuals (Keil et al, 2012). More recent studies have indicated that beyond a specific region of interest, a large-scale network of oscillatory brain networks are involved in effectuating cross-modal perception(Kumar et al, 2016). A key question emerges how robust is this network across a group of individuals and whether the organization of these networks contingent on the stimulus configurations or the perceptual outcome, specifically in the case of McGurk incongruent stimulus.…”

Multiscale neural connectivity during human sensory processing in the brain

“…Many studies have shown the relationship between the electrical activity of brain and complex psychophysiological processes such as alertness [1], arousal [2], attention [3], memory [4], and executive functions [5]. Among these tasks, analysis of brain activity related to processing of visual stimuli is one of the most interesting and actively studied in modern science [6][7][8].…”

Lifespan associated global patterns of coherent neural communication

“…To measure the covariation of neural oscillations on a global level, we employed a technique known as global coherence (Cimenser et al, 2008;Kumar et al,2016). Global coherence among sensors at any frequency f is measured as the percentage of variance explained by the first eigenvector of the cross spectral density matrix at f .…”