Visual Recalibration of Auditory Speech Identification

Abstract

The kinds of aftereffects, indicative of cross-modal recalibration, that are observed after exposure to spatially incongruent inputs from different sensory modalities have not been demonstrated so far for identity incongruence. We show that exposure to incongruent audiovisual speech (producing the well-known McGurk effect) can recalibrate auditory speech identification. In Experiment 1, exposure to an ambiguous sound intermediate between /aba/ and /ada/ dubbed onto a video of a face articulating either /aba/ or /ada/ increased the proportion of /aba/ or /ada/ responses, respectively, during subsequent sound identification trials. Experiment 2 demonstrated the same recalibration effect or the opposite one, fewer /aba/ or /ada/ responses, revealing selective speech adaptation, depending on whether the ambiguous sound or a congruent nonambiguous one was used during exposure. In separate forced-choice identification trials, bimodal stimulus pairs producing these contrasting effects were identically categorized, which makes a role of postperceptual factors in the generation of the effects unlikely.

Perceptual learning for speech: Is there a return to normal?

“…In both studies, only top-down constraint from the word contexts could be producing the identification of the /b/ or /d/ (Samuel, 1997) or the /s/ or /S/ (Samuel, 2001), with these phonemes in turn producing the observed adaptation shifts. Critically, as Bertelson et al (2003) have shown, adaptation shifts are in the opposite direction from those found through perceptual learning. For example, perceptual learning due to exposure to items like ''arthriti?''…”

“…The distinctiveness of these effects has been elegantly demonstrated in a study by Bertelson et al (2003), and in followup work by that group (Vroomen et al, submitted;Vroomen, van Linden, Keetels, de Gelder, & Bertelson, 2004). In these studies the perceptual learning is driven by audiovisual manipulations, rather than lexicality, but the effects seem comparable.…”

“…In the large and well-developed literature on adaptation, a study by Harris (1980) suggests that the categorization shifts fade within approximately a half an hour. Thus, even though perceptual learning and selective adaptation are different phenomena (Bertelson et al, 2003;Vroomen, van Linden, de Gelder, & Bertelson, submitted) the adaptation results provide a reasonable starting point for our investigation.…”

“…Rather than align the signal to match the representations, the representations can be adjusted to align with the input. A burgeoning literature on such perceptual learning effects for speech suggests that this approach is promising (e.g., Bertelson et al, 2003;Eisner & McQueen, 2005;Kraljic & Samuel, in press;Maye et al, 2003;Norris et al, 2003;Vroomen et al, submitted).…”

“…Norris, McQueen, and Cutler (2003) demonstrated that listeners are able to adjust (retune) their phonetic representations according to the speech they hear with surprisingly little exposure; just 20 of the same ÔoddlyÕ pronounced phonemes embedded somewhere in 200 words will lead to perceptual learning. Bertelson, Vroomen, and de Gelder (2003) showed that such learning can also be driven by a visual context that disambiguates the ÔoddÕ phoneme (they call the effect recalibration). Maye et al (2003) found that listening to accented English speech results in phonemic categories that are adjusted in the specific direction of the accent, and not simply in more broad or sloppy categories.…”