Two experiments test whether isolated visible speech movements can be used for face matching. Visible speech information was isolated with a point-light methodology. Participants were asked to match articulating point-light faces to a fully illuminated articulating face in an XAB task. The first experiment tested single-frame static face stimuli as a control. The results revealed that the participants were significantly better at matching the dynamic face stimuli than the static ones. Experiment 2 tested whether the observed dynamic advantage was based on the movement itself or on the fact that the dynamic stimuli consisted of many more static and ordered frames. For this purpose, frame rate was reduced, and the frames were shown in a random order, a correct order with incorrect relative timing, or a correct order with correct relative timing. The results revealedbetter matching performance with the correctly ordered and timed frame stimuli, suggesting that matches were based on the actual movement itself. These findings suggest that speaker-specific visible articulatory style can provide information for face matching.

Three experiments examined whether image manipulations known to disrupt face perception also disrupt visual speech perception. Research has shown that an upright face with an inverted mouth looks strikingly grotesque whereas an inverted face and an inverted face containing an upright mouth look relatively normal. The current study examined whether a similar sensitivity to upright facial context plays a role in visual speech perception. Visual and audiovisual syllable identification tasks were tested under 4 presentation conditions: upright face-upright mouth, inverted face-inverted mouth, inverted face-upright mouth, and upright face-inverted mouth. Results revealed that for some visual syllables only the upright face-inverted mouth image disrupted identification. These results suggest that upright facial context can play a role in visual speech perception. A follow-up experiment testing isolated mouths supported this conclusion.

The effects of talker variability on visual speech perception were tested by having subjects speechread sentences from either single-talker or mixed-talker sentence lists. Results revealed that changes in talker from trial to trial decreased speechreading perfonnance. To help determine whether this decrement was due to talker change-and not a change in superficial characteristics of the stimuliExperiment 2 tested speechreading from visual stimuli whose images were tinted by a single color, or mixed colors. Results revealed that the mixed-color lists did not inhibit speechreading performance relative to the single-color lists. These results are analogous to findings in the auditory speech literature and suggest that, like auditory speech, visual speech operations include a resource-demanding component that is influenced by talker variability.

No abstract

Seeing a speaking face can influence observers’ auditory perception of syllables [McGurk and McDonald, Nature 264, 746–748 (1976)]. This effect decreases when the speaker’s face is inverted [e.g., Green, 3014 (1994)]. Face recognition is also inhibited with inverted faces [e.g., Rock, Sci. Am. 230, 78–85 (1974)] suggesting a similar underlying process. To further explore the link between face and audiovisual speech perception, a speech experiment was designed to replicate another face perception effect. In this effect, an inverted face and an inverted face containing upright lips are perceived as looking normal, but an upright face with inverted lips looks grotesque [Thompson, Perception 9, 438–484 (1980)]. An audiovisual speech experiment tested four presentation conditions: Upright face-upright mouth, upright face-inverted mouth, inverted face-inverted mouth, inverted face-upright mouth. Various discrepant audiovisual syllables were tested in each condition. Visual influences occurred in all but the upright face-inverted mouth condition for some of the syllable combinations thereby mimicking the face perception effect. However, other syllable combinations revealed visual influences in all four conditions. Results are interpreted in terms of articulatory dynamics and the vertical symmetry of the visual stimuli.