The effect of face and lip inversion on audiovisual speech integration

Abstract

Seeing a speaking face can influence observers’ auditory perception of syllables [McGurk and McDonald, Nature 264, 746–748 (1976)]. This effect decreases when the speaker’s face is inverted [e.g., Green, 3014 (1994)]. Face recognition is also inhibited with inverted faces [e.g., Rock, Sci. Am. 230, 78–85 (1974)] suggesting a similar underlying process. To further explore the link between face and audiovisual speech perception, a speech experiment was designed to replicate another face perception effect. In this effect, an inverted face and an inverted face containing upright lips are perceived as looking normal, but an upright face with inverted lips looks grotesque [Thompson, Perception 9, 438–484 (1980)]. An audiovisual speech experiment tested four presentation conditions: Upright face-upright mouth, upright face-inverted mouth, inverted face-inverted mouth, inverted face-upright mouth. Various discrepant audiovisual syllables were tested in each condition. Visual influences occurred in all but the upright face-inverted mouth condition for some of the syllable combinations thereby mimicking the face perception effect. However, other syllable combinations revealed visual influences in all four conditions. Results are interpreted in terms of articulatory dynamics and the vertical symmetry of the visual stimuli.

Les illusions McGurk dans la parole : 25 ans de recherches

“…Lorsque /ba/ auditif était doublé de mouvements articu latoires synthétiques correspondant à /ga/, aucun biais visuel n'était observé. Par contre, le doublage inverse (/ga/ auditif sur /ba/ visuel) don nait bien lieu à des combinaisons (Massaro, 1998b (Jordan et Bevan, 1997) ou pré senté à l'envers (Bertelson et al, 1994 ;Green, 1994 ;Yakel, Rosenblum, Green, Bosley et Vasquez, 1995 ;Jordan et Bevan, 1997 ;Colin et al, 2001). L'influence de la présentation inversée des stimuli visuels résulte probablement de la modification de la configuration spatiale des articulateurs .…”