Face and mouth inversion effects on visual and audiovisual speech perception.

Abstract

Three experiments examined whether image manipulations known to disrupt face perception also disrupt visual speech perception. Research has shown that an upright face with an inverted mouth looks strikingly grotesque whereas an inverted face and an inverted face containing an upright mouth look relatively normal. The current study examined whether a similar sensitivity to upright facial context plays a role in visual speech perception. Visual and audiovisual syllable identification tasks were tested under 4 presentation conditions: upright face-upright mouth, inverted face-inverted mouth, inverted face-upright mouth, and upright face-inverted mouth. Results revealed that for some visual syllables only the upright face-inverted mouth image disrupted identification. These results suggest that upright facial context can play a role in visual speech perception. A follow-up experiment testing isolated mouths supported this conclusion.

Visual speech information for face recognition

“…Stimulus manipulations known to disproportionatelydisrupt face recognition have also been shown to inhibit visual speech perception (see Rosenblum et al, 2000, for a review). This has been found of inverted faces (Green, 1994;Jordan & Bevan,1997;Massaro & Cohen, 1996), as well as of faces in which the mouth alone is inverted (i.e., the Margaret Thatcher effect; Rosenblum et al, 2000;Thompson, 1980). The fact that both functions are influenced by these specific manipulations has been interpreted as support for the involvement of common information or processes (e.g., Jordan & Bevan, 1997;Rosenblum et al, 2000).…”

“…When speech is seen, the functions of face recognition and lipreading are also considered to be distinct in information, operations, and neurophysiology (Bruce & Young, 1986;Campbell, Landis, & Regard, 1986;Ellis, 1989;Fodor, 1983;Liberman & Mattingly, 1985). However, recent research on both modalities has challenged these assumed independencies(for reviews, see Nygaard, Sommers, & Pisoni, 1994;Remez, Fellowes, & Rubin, 1997;Rosenblum, Yakel, & Green, 2000;Schweinberger & Soukup, 1998;Yakel, Rosenblum, & Fortier, 2000). Regarding auditory speech, there is evidence that voice information can facilitate linguistic recovery in the contexts of word naming and identification in noise (Mullennix, Pisoni, & Martin, 1989;Nygaard et al, 1994), recognition memory (Craik & Kirsner, 1974;Palmeri, Goldinger, & Pisoni, 1993), implicit memory (Church & Schacter, 1994), and form-based priming (Saldaña & Rosenblum, 1994).…”

When half a face is as good as a whole: Effects of simple substantial occlusion on visual and audiovisual speech perception

“…Each utterance comprised a consonant-/b/, /g/, or /v/-plus one of two vowels-/a/ or /i/; these syllables are known to produce powerful effects of visual speech on auditory speech perception when presented with whole-face stimuli (e.g., Green, Kuhl, & Meltzoff, 1988;Jordan & Bevan, 1997;Jordan & Thomas, 2001;Rosenblum et al, 2000). Each articulation began and ended with the face in neutral repose (mouth closed), and each syllable was spoken naturally with no artificial emphasis on articulation.…”

“…The facial movements that accompany speech production (visual speech) are a powerful component of speech perception (e.g., Erber, 1969;Jordan, McCotter, & Thomas, 2000;MacLeod & Summerfield, 1987, 1990McGurk & MacDonald, 1976;Middleweerd & Plomp, 1987;Reisberg, McLean, & Goldfield, 1987;Rosenblum, Yakel, & Green, 2000;Sumby & Pollack, 1954;Thomas & Jordan, 2002). In particular, seeing the articulating face of a talker can improve auditory speech intelligibility substantially in quiet and noisy environments, and in the McGurk effect (after McGurk & MacDonald, 1976), can alter the perceived identity of speech sounds.…”

Energy, information, detection, and action

“…Many Gibsonian-based explanations of multi-modal effects -including our own (e.g., Rosenbium & Saldaña 1996;Rosenblum et al 2000) -have provided only a rough sketch of how specification might be instantiated cross-modally. S&B's thoughtful examination of these issues provides an invaluable step toward solving the problem.…”