Audiovisual presentation demonstrates that selective adaptation in speech perception is purely auditory

Abstract

. Both auditory and phonetic processes have been implicated by previous results from selective adaptation experiments using speech stimuli. It has proved difficult to dissociate their individual contributions because the auditory and phonetic structure of conventional acoustical stimuli are mutually predictive. In the present experiment, the necessary dissociation was achieved by using an audiovisualadaptor consisting of an acoustical [bE] synchronized to a video recording of a talker uttering the syllable [gEl. This stimulus was generallyidentifiedas oneof the dentals [dE] or [a E]. It producedan adaptation effect,measuredwith an acoustical[be-ds] test continuum, identical in size and direction to that produced by an acoustical [bEl-an adaptor sharing its acoustical structure-and opposite in direction to that produced by an acoustical [dEl-an adaptor sharing its perceived phonetic identity. Thus, the result strongly suggests that auditory rather than phonetic levels of processing are influenced in selective adaptation.Since the introduction of the selective adaptation paradigm to research on speech perception (Eimas & Corbit, 1973), many questions have been raised concerning the nature and loci of the processes underlying the effects obtained. The paradigm is straightforward. A series of speech sounds varying in some distinctive acoustic parameter is created, usually spanning two phonetic categories. These test syllables are randomized and presented for identification under two conditions. In the baseline condition, the individual syllables are presented for identification in isolation. In the adaptation condition, each test syllable is presented following a number of repetitions of an adapting syllable. In each condition, an ogive is fitted to the identification functions from several presentations of each syllable in a series. The point of response equiprobability, the phoneme boundary, is determined. The position of the boundary in the series usually changes following adaptation, and the size of the change provides a measure of the adaptation effect. When the adaptor is an unambiguous syllablecorrespondingto one or the other of the endpoints of the test continuum, the boundary usually moves toward that syllable. If, for instance, the test series embraces a phonetic contrast in voicing (e.g., [ba-pha]), following adaptation with [ba] a greater proportion of test syllables, particularly in the boundary region, are identified as [phal, and the phoneme boundary shifts toward the [ba] end of the series.Originally, it was proposed (Cooper, 1974;

Phonetic Recalibration in Audiovisual Speech

“…Others however (e.g., Sawusch, 1977), showed that the size of selective speech adaptation depends upon the degree of spectral overlap between the adapter and test sound, and that most, if not all of the effect is auditory rather than phonetic. A similar conclusion was reached by Roberts and Summerfield (1981). They exposed listeners to audiovisual congruent (auditory /b/ with lipread /b/) or incongruent adapter stimuli (auditory /b/ with lipread /g/) and obtained similar aftereffects, despite the fact that the adapters were perceived differently.…”

“…For example, Roberts and Summerfield (1981) demonstrated that adaptation was induced by the auditory component, whereas the phonetic label attached to the adapting stimulus had no effect. Here we also observed that equal amounts of selective adaptation were obtained for listeners in speech or non-speech mode.…”

“…These stimuli were not expected to induce recalibration because there is no deviance between sight and sound that supposedly drives recalibration. However, the unambiguous nature of the lipread stimuli might possibly cause a contrastive aftereffect (in the auditory domain known as 'selective speech adaptation') as has been demonstrated before for auditory speech (Roberts & Summerfield, 1981;Saldaña & Rosenblum, 1994), color, curvature (Gibson, 1933, or motion (Anstis, 1986), possibly reflecting a 'fatigue' of some hypothetical feature detectors. With unambiguous audiovisual exposure stimuli, one might thus expect fewer 'n'-responses after exposure to VnAn than VmAm, an effect in the opposite direction of recalibration.…”

“…If listeners are in speech mode, heard and lipread inputs are combined into a single phonetic representation, but not so if listeners are not under the impression that the auditory and visual signals refer to separate events. Selective adaptation, though, may occur for listeners in speech and non-speech mode, assuming that this phenomenon depends on some low-level acoustic factor and not the phonetic interpretation of the sound (Roberts & Summerfield, 1981).…”

Why not model spoken word recognition instead of phoneme monitoring?

“…Samuel's results contrast with those of Roberts and Summerfield (1981) and Saldaña and Rosenblum (1994), who used the McGurk effect (McGurk & McDonald 1976) to investigate whether adaptation was driven by the acoustic form of the input or by its phonemic percept. The Saldaña and Rosenblum study took advantage of the fact that an auditory /ba/ presented with a video of a spoken /va/ is perceived as /va/.…”