Studies of the McGurk effect: implications for theories of speech perception

Abstract

Studies of the McGurk effect demonstrate that observers integrate auditory information with visual information from a talker's face during speech perception. The findings from these studies pose challenges for theories of speech perception that must account for how and why the auditory and visual information are integrated. One theoretical issue concerns the objects of speech perception. Some researchers claim that the objects of speech perception are articulatory gestures while others argue that the objects are auditory in nature. The McGurk effect is often taken as evidence for gestural approaches because such theories provide a good account for why the auditory and visual information are integrated during perception. The findings from various studies of the McGurk effect including cross-modal context effects, developmental influences, and neuromagnetic measures of brain activation will be reviewed. The implication of these findings will be discussed with regard to whether the metric for combining the auditory and visual information is best thought of as auditory or gestural in nature.

Effects of speechreading and signal-to-noise ratio on understanding mainstream American English by American and Indian adults.

“…If so, why are visual cues combined with auditory cues when auditory cues are available to accurately understand speech? These psycholinguists theorized the purpose of speech perception was whether they are auditory related or articulatory related in nature (Green, 1996). Cognitive psychologists also proposed auditory and articulatory theories to explain auditory-visual integration in speech perception.…”

“…Thai language contains the phoneme /ŋ/ in both word-initial and word-final positions, but English does not contain the phoneme /ŋ/ in word-initial position. Thus, Burnham (1998) hypothesized that English participants would respond more with /n/ in the initial position than in the final position due to visual ambiguity of /ŋ/ (visual /ŋ/ looks like visual /n/) and their language experience (Burnham, 1998 Green (1996) concluded that the McGurk effect occurs phonetically (i.e., vowel context) not phonologically (listener's language). The cross-language studies revealed that this auditory-visual integration occurs at an earlier phonetic level of processing and that integration is affected by phonetic factors (Green, 1996).…”

For speech perception by humans or machines, three senses are better than one

“…Other related studies followed the initial description of the McGurk illusion [e.g., 2,3], and several different theories were offered [4,5,6]. Green [2], in this session, presents an overview of research on the McGurk illusion.…”

“…Green [2], in this session, presents an overview of research on the McGurk illusion. The paper by Sekiyama et al [3], in this session, shows that the illusion is sensitive to native language, acoustic quality, language proficiency, and cultural factors.…”

“…Possibilities include the sounds of speech [26], the articulatory gestures [5], the talker's linguistic intentions [4], and the linguistic units of language [27]. Green [2], in this session, points out that the McGurk illusion is frequently interpreted as evidence that the common metric is articulatory, and he reviews other phenomena that support either an articulatory or auditory metric.…”