Language/Culture Modulates Brain and Gaze Processes in Audiovisual Speech Perception

Abstract

Several behavioural studies have shown that the interplay between voice and face information in audiovisual speech perception is not universal. Native English speakers (ESs) are influenced by visual mouth movement to a greater degree than native Japanese speakers (JSs) when listening to speech. However, the biological basis of these group differences is unknown. Here, we demonstrate the time-varying processes of group differences in terms of event-related brain potentials (ERP) and eye gaze for audiovisual and audio-only speech perception. On a behavioural level, while congruent mouth movement shortened the ESs’ response time for speech perception, the opposite effect was observed in JSs. Eye-tracking data revealed a gaze bias to the mouth for the ESs but not the JSs, especially before the audio onset. Additionally, the ERP P2 amplitude indicated that ESs processed multisensory speech more efficiently than auditory-only speech; however, the JSs exhibited the opposite pattern. Taken together, the ESs’ early visual attention to the mouth was likely to promote phonetic anticipation, which was not the case for the JSs. These results clearly indicate the impact of language and/or culture on multisensory speech processing, suggesting that linguistic/cultural experiences lead to the development of unique neural systems for audiovisual speech perception.

Age-Related Alterations in Audiovisual Integration: A Brief Overview

“…Using the same technique, Arnal et al (2009) reported an amplitude reduction and latency shortening in the M100A peak in the audiovisual condition relative to that in the auditory-only condition, which indicated AVI in audiovisual condition (Arnal et al, 2009). A similar result-that amplitude reduction in the N1 and P2 components was elicited during audiovisual stimuli processing relative to auditory-only stimuli processing-was also reported in studies using ERPs in an auditory/visual detection task (Vroomen & Stekelenburg, 2010), an action perception task (Stekelenburg & Vroomen, 2007), and a speech perception task (Hisanaga et al, 2016;Pilling, 2009;Stekelenburg & Vroomen, 2007). The authors presumed that the accompanying visual information assists the participant in identifying the stimuli by reducing signal uncertainty and lowering computational demands for auditory brain areas, leading to a deactivation in audiovisual information processing.…”

A value-driven McGurk effect: Value-associated faces enhance the influence of visual information on audiovisual speech perception and its eye movement pattern

“…Adding a concurrent cognitive task to the main McGurk task would decrease the McGurk proportion as well as the time looking at the mouth area (Buchan & Munhall, 2012). In addition, weak McGurk perceivers (i.e., perceivers who perceive the McGurk effect less frequently in general) fixate less on the talker's mouth area compared with strong McGurk perceivers (Gurler, Doyle, Walker, Magnotti, & Beauchamp, 2015;Hisanaga, Sekiyama, Igasaki, & Murayama, 2016).…”

Random, Scattered and Asymmetric Distribution of Fixational Eye Movement - Experimental Evidence

“…So, it is easy to find the eye movement or micro saccades associated with neurons' active status. Of course, cognitive psychologists have now identified eye movement study is meaningful and have used eye movements for research in areas such as reading and mental development, and many results have been achieved [17,18,19,20,21].…”

“…We pay special attention to the research results mentioned above, as well as the study of eye movement on parafoveal processing in reading and linguistics [19,20,21]. However, in terms of the exact role of eye movements in visual information processing, there are very few published related research results, so there is still a need for further discussion (see the Discussion section).…”