Audiovisual integration in perception of real words

Abstract

Three experiments follow up on Easton and Basala's (1982) report that the "McGurk effect" (an influence of a visibly mouthed utterance on a dubbed acoustic one) does not occur when utterances are real words rather than nonsense syllables. In contrast, with real·word stimuli, Easton and Basala report a strong reverse effect whereby a dubbed soundtrack strongly affects identification of lipread words. In Experiment 1, we showed that a strong McGurk effect does obtain when dubbed real words are discrepant with observed words in consonantal place of articulation. A second experiment obtained only a weak reverse effect of dubbed words on judgments of lipread words. A final experiment was designed to provide a sensitive test of effects of lipread words on judgments of heard words and of heard words on judgments of lipread words. The findings reinforced those of the first two experiments that both effects occur, but, with place-of-articulation information discrepant across the modalities, the McGurk effect is strong and the reverse effect weak.In the "McGurk effect," first observed by McGurk and MacDonald (1976; see also MacDonald & McGurk, 1978), a listener's perceptual report of a heard nonsense syllable can be influenced by the sight of a speaker producing a different syllable. In one experiment, MacDonald and McGurk dubbed a videotape in which a speaker produced various consonant-vowel (CV) syllables. On the dubbed tape, the mouthed syllable and the syllable on the soundtrack were discrepant. Syllables that subjects reported hearing were influenced by the mouthed syllable.For example, mouthed ldal paired with acoustic lmal was most frequently identified as "na." The manner-place hypothesis put forth by MacDonald and McGurk seems to account for much of the data (but see Summerfield, 1987, for some qualifications). The hypothesis suggests that the acoustic signal dominates in perception of manner of articulation (and voicing) where optical information is poor or absent. In contrast, optical information dominates in perception of those places of articulation that are visibly apparent.Although an effect of visual information was not anticipated by any theory of speech perception, a natural accounting of it is available in at least three current theories.

Hemispheric contributions to the integration of visual and auditory information in speech perception

“…On average, the right-handed subjects selected the McGurk alternative on 79% of trials. This percentage is identical to that reported by Dekle et al (1992), who used the same dubbed words, but without lateralizing them. 3 Figure 1 (top) shows the percentages of McGurk responses among our right-handed subjects, plotted separately for the two visual fields of face and word presentation.…”

“…Stimulus materials were derived from those used by Dekle, Fowler, and Funnell (1992). The nine trial types listed in Table 1 Each trial in the practice and test sets consisted of the following sequence: an arrow indicated direction of fixation, a dubbed audiovisual word was presented, a second arrow indicated direction of fixation, and then a printed word pair was presented.…”

Cultural and linguistic factors in audiovisual speech processing: The McGurk effect in Chinese subjects

“…While this effect has been shown to be robust in English-speaking cultures (e.g., Dekle, Fowler, & Funnell, 1992;Green, Kuhl, Meltzoff, & Stevens, 1991;Massaro & Cohen, 1983;Rosenblum & Saldana, 1992), we have found interlanguage differences between native speakers ofJapanese and native speakers of American English. In our first study, native speakers of Japanese hardly showed the McGurk effect when listening to very clear Japanese speech, although these Japanese subjects showed a highly increased McGurk effect when auditory noise was added to the stimuli (Sekiyama & Tohkura, 1991).…”

Toward Perceptually Realistic Talking Heads: Models, Methods, and McGurk

“…These were chosen through pilot testing from a larger collection of monosyllabic words taken from past research into the McGurk effect [Dekle et al 1992] [Dodd 1977] [Easton and Basala 1982]. Table 1 gives the tuples which were chosen.…”

“…The McGurk effect (as the phenomena is now widely known) has been replicated several times [MacDonald et al 2000] [Dekle et al 1992] [Dodd 1977] using varieties of visual and auditory stimuli. An interesting summary of expected misinterpreted audio syllables, given the influence of a differing visual syllable, may be found in [Dodd 1977].…”