Auditory–visual speech integration by prelinguistic infants: Perception of an emergent consonant in the McGurk effect

Abstract

The McGurk effect, in which auditory [ba] dubbed onto [ga] lip movements is perceived as "da" or "tha," was employed in a real-time task to investigate auditory-visual speech perception in prelingual infants. Experiments 1A and 1B established the validity of real-time dubbing for producing the effect. In Experiment 2, 4 1/2-month-olds were tested in a habituation-test paradigm, in which an auditory-visual stimulus was presented contingent upon visual fixation of a live face. The experimental group was habituated to a McGurk stimulus (auditory [ba] visual [ga]), and the control group to matching auditory-visual [ba]. Each group was then presented with three auditory-only test trials, [ba], [da], and [(delta)a] (as in then). Visual-fixation durations in test trials showed that the experimental group treated the emergent percept in the McGurk effect, [da] or [(delta)a], as familiar (even though they had not heard these sounds previously) and [ba] as novel. For control group infants [da] and [(delta)a] were no more familiar than [ba]. These results are consistent with infants' perception of the McGurk effect, and support the conclusion that prelinguistic infants integrate auditory and visual speech information.

A comparison of the McGurk effect for spoken and sung syllables

“…In particular, there were relatively fewer / / responses in the out-of-key condition. Audiovisual speech research has also revealed differences in the proportion of / / versus / / responses, although these differences are variable in extent and are poorly understood (Burnham & Dodd, 2004;Green, Kuhl, Meltzoff, & Stevens, 1991;Massaro & Cohen, 1983). The type of fused response selected appears to depend on a range of factors, including the speaker, vowel type, whether visual cues were live or recorded, and how auditory stimuli were synthesized.…”

The processing of audio-visual speech: empirical and neural bases

“…The combinatorial rules and processes involved have exercised psychologists for many years (see Bernstein et al 2004a for review), but this work has focused on processing at the phoneme level. The findings that phonetic context perceived by eye can shift phonemic category boundaries (Green et al 1991), and that prelinguistic babies are sensitive to McGurk effects (Burnham & Dodd 2004), demonstrate that audio-visual integration can occur 'pre-phonemically'. That said, the phonemic level of linguistic structure offers the most approachable entry point for examining many aspects of the perception of seen speech in the absence of hearingthat is, silent speech-reading.…”

“…Infants who are not yet able to speak or respond to attentional instruction are sensitive to audio-visual fusions. From the age of six months or so, infants who have habituated to a 'McGurk' audiovisual 'da' (the stimulus comprises an auditory 'ba' dubbed to a visual 'ga') dishabituate when a congruent audio-visual 'ba' is played to them, and fail to respond with a dishabituation response when a 'real' audiovisual 'da', derived from a visual 'da' and an auditory 'da', is played to them (Burnham & Dodd (2004) and see also Rosenblum et al (1997)). Audio-visual speech processing capabilities are in place even before the child has useful speech.…”

Behavioral and Neural Foundations of Multisensory Face-Voice Perception in Infancy

“…For example, infants and adults alike are susceptible to the McGurk illusion (e.g., Burnham & Dodd, 2004;McGurk & MacDonald, 1976;Rosenblum, Schmuckler, & Johnson, 1997). Specifically, infants (and adults) commonly Intersensory Perception 12 perceive a unique or third speech event when one auditory speech sound is artificially synchronized with different visible speech movements (i.e., lip movements) (Burnham & Dodd, 2004;Rosenblum et al, 1997). In addition to integrating different face and voice speech events, as in the case of the McGurk illusion, infants' also unite auditory and visual speech events in the absence of spatial contiguity (e.g., Pons, Lewkowicz, Soto-Faraco, & Sebastian-Galles, 2009).…”