Listening with eye and hand: Cross-modal contributions to speech perception.

Abstract

Three experiments investigated the basis for the-McGurk effect" whereby opticallyspecified syllables experienced synchronously with acoustically-specified syllables integrate in perception to determine a listener's auditory perceptual experience. One hypothesis is that the effect arises when optical and acoustic cues for a syllable are associated in memory. A second hypothesis is that the effect arises when cross-modal information, familiar or not, is convincingly about the same articulatory speech event in the environment. Experiments contrasted the cross-modal effect of orthographic syllables on acoustic syllables, presumed to be associated in experience and memory, with that of haptically-experienced and acoustic syllables, presumed not to be associated. Findings were that the latter pairing, but not the former, gave rise to cross-modal influences under conditions in which subjects were informed that cross-modal syllables were paired independently. Mouthed syllables felt by a perceiver affected reports of simultaneously heard syllables (and reports of the mouthed syllables were affected by the heard syllable). These effects were absent when syllables were simultaneously seen (spelled) and heard. We conclude that the McGurk effect does not arise due to association in memory, but rather to conjoint near specification of the same causal source in the environment..in speech, the moving vocal tract producing phonetic gestures.

The Role of Lip-reading and Cued Speech in the Processing of Phonological Information in French-educated Deaf Children

“…This class of model implies that the speech processing system accepts a combination of``exotic'' signals (Cues) and natural ones (lip-reading and residual hearing). Some results by Fowler and Dekle (1991) suggest that this type of interaction is possible, at least under certain conditions. They demonstrated that auditory and tactile lip information (i.e.…”

The processing of audio-visual speech: empirical and neural bases

“…Summerfield (1987) suggested that when we perceive speech, we reconstruct the patterns of articulation used by the talker, irrespective of the modality of input. Visual and even haptic processes (Fowler & Dekle 1991) can affect the impression of what was heard. While information from these modalities may be integrated with acoustic information via purely associative mechanisms, it seems probable that the processing system will make use of the correspondences between visual, somaesthetic and acoustic events to inform processing.…”

Facial identity and facial speech processing: Familiar faces and voices in the McGurk effect

“…Since this initial demonstration of the McGurk effect, further studies have supported and extended the phenomenon and have sought to offer explanations for its existence (e.g., see Fowler & Dekle, 1991;MacDonald & McGurk, 1978;Massaro & Cohen, 1983;Summerfield, 1979). The effect appears to be extremely robust under a variety ofconditions: It is not lessened by the perceiver having prior knowledge ofthe illusion, nor is it decreased when the perceiver has had considerable practice at selectively attending (McGurk & MacDonald, 1976); further, the effect remains when subjects are specifically requested to report only what they have heard (Summerfield & McGrath, 1984).…”