Cultural and linguistic factors in audiovisual speech processing: The McGurk effect in Chinese subjects

Sekiyama, Kaoru

doi:10.3758/bf03206849

Cited by 131 publications

(153 citation statements)

References 15 publications

Supporting

Mentioning

143

Contrasting

Unclassified

Order By: Relevance

“…In addition, the employment of non-native language visual cues in the perception of the target contrasts by the experimental group is at odds with the hypothesis that tone language speakers are less likely to use visual information in non-native speech perception/production (Sekiyama, 1997;Sekiyama and Tohkura, 1993). In previous studies, Mandarin speakers showed a relatively lower degree of use of visual information in speech perception than non-tone language speakers (de Gelder and Vroomen, 1992).…”

Section: The Effect Of Articulatory Information On the Subjects' Percmentioning

confidence: 58%

“…In previous studies, Mandarin speakers showed a relatively lower degree of use of visual information in speech perception than non-tone language speakers (de Gelder and Vroomen, 1992). It has been argued that since Mandarin is a tone language, L1-Mandarin speakers rely more on tones than on visual cues in speech perception (Sekiyama, 1997;Sekiyama and Tohkura, 1993). Moreover, Hazan et al (2006) indicated that L2 listeners may lose sensitivity to visemes that do not exist in their L1.…”

Section: The Effect Of Articulatory Information On the Subjects' Percmentioning

confidence: 99%

See 1 more Smart Citation

Audiovisual Training Effects on L2 Speech Perception and Production

Liu

2016

IJELT

View full text Add to dashboard Cite

This study investigated whether audiovisual perception training can improve learners' auditory perception of L2 speech sounds. 29 subjects (experimental group) who had difficulty in the perception of English /θ/-/s/ and /ð/-/z/ were recruited to attend a 9-session audiovisual perception training programme with identification tasks on the target contrasts. Their perception performance was tested before, during and at the end of the training with an AXB task. A control group was tested with the same AXB task and intervals as that of the experimental group. The results showed that the experimental group's perception accuracy improved substantially during and by the end of the training programme. The control group also showed perception improvement across the pre-test and post-test. Their degree of improvement, however, was significantly lower than that of the experimental group. These results therefore confirm the value of the audiovisual modality in L2 speech perception training.

show abstract

Section: The Effect Of Articulatory Information On the Subjects' Percmentioning

confidence: 58%

Section: The Effect Of Articulatory Information On the Subjects' Percmentioning

confidence: 99%

Audiovisual Training Effects on L2 Speech Perception and Production

Liu

2016

IJELT

View full text Add to dashboard Cite

show abstract

“…However, the reverse combination of an auditory nonlabial, /ka/ on a visual labial, /pa/ is known to hardly induce such an illusory fusion, resulting in a correct auditory percept, especially in Japanese participants. This is thought to be due to either the sound structure of Japanese or their nature of less reliance on visual information in speech perception (Sekiyama, 1997). In such a lack of audio-visual speech fusion, they are often aware of the discrepancy in audio-visual speech information, because the visual /pa/ (labial) and the correctly perceived voice, /ka/ (non-labial) is different with respect to the place of articulation.…”

Section: Conditionsmentioning

confidence: 99%

Perceptual congruency of audio-visual speech affects ventriloquism with bilateral visual stimuli

Kanaya

Yokosawa

2010

Psychon Bull Rev

View full text Add to dashboard Cite

Many studies on multisensory processes have focused on performance in simplified experimental situations, with a single stimulus in each sensory modality. However, these results cannot necessarily be applied to explain our perceptual behavior in natural scenes where various signals exist within one sensory modality. We investigated the role of audio-visual syllable congruency on participants' auditory localization bias or the ventriloquism effect using spoken utterances and two videos of a talking face. Salience of facial movements was also manipulated. Results indicated that more salient visual utterances attracted participants' auditory localization. Congruent pairing of audio-visual utterances elicited greater localization bias than incongruent pairing, while previous studies have reported little dependency on the reality of stimuli in ventriloquism. Moreover, audio-visual illusory congruency, owing to the McGurk effect, caused substantial visual interference on auditory localization. Multisensory performance appears more flexible and adaptive in this complex environment than in previous studies.

show abstract

“…The McGurk effect is a striking phenomenon under incoherent AV information, where lip-read information in speech interferes with auditory perception of phonemes (McGurk & MacDonald 1976;Sekiyama & Tohkura 1991;Munhall et al 1996;Sekiyama 1997), suggesting a strong influence of vision (figure 1a). There are two typical responses in the McGurk effect.…”

Section: Introductionmentioning

confidence: 99%

Fusion and combination in audio-visual integration

Omata

Masamune

2007

Proc. R. Soc. A.

View full text Add to dashboard Cite

Language is essentially multi-modal in its sensory origin, the daily conversation depending heavily on the audio-visual (AV) information. Although the perception of spoken language is primarily dominated by audition, the perception of facial expression, particularly that of the mouth, helps us comprehend speech. The McGurk effect is a striking phenomenon where the perceived phoneme is affected by the simultaneous observation of lip movement, and probably reflects the underlying AV integration process. The elucidation of the principles involved in this unique perceptual anomaly poses an interesting problem. Here we study the nature of the McGurk effect by means of neural networks (self-organizing maps, SOM) designed to extract patterns inherent in audio and visual stimuli. It is shown that a McGurk effect-like classification of incoming information occurs without any additional constraint or procedure added to the network, suggesting that the anomaly is a consequence of the AV integration process. Within this framework, an explanation is given for the asymmetric effect of AV pairs in causing the McGurk effect (fusion or combination) based on the 'distance' relationship between audio or visual information within the SOM. Our result reveals some generic features of the cognitive process of phoneme perception, and AV sensory integration in general.

show abstract

Cultural and linguistic factors in audiovisual speech processing: The McGurk effect in Chinese subjects

Cited by 131 publications

References 15 publications

Audiovisual Training Effects on L2 Speech Perception and Production

Audiovisual Training Effects on L2 Speech Perception and Production

Perceptual congruency of audio-visual speech affects ventriloquism with bilateral visual stimuli

Fusion and combination in audio-visual integration

Contact Info

Product

Resources

About