Contributions of Oral and Extraoral Facial Movement to Visual and Audiovisual Speech Perception.

Abstract

Seeing a talker's face influences auditory speech recognition, but the visible input essential for this influence has yet to be established. Using a new seamless editing technique, the authors examined effects of restricting visible movement to oral or extraoral areas of a talking face. In Experiment 1, visual speech identification and visual influences on identifying auditory speech were compared across displays in which the whole face moved, the oral area moved, or the extraoral area moved. Visual speech influences on auditory speech recognition were substantial and unchanging across whole-face and oral-movement displays. However, extraoral movement also influenced identification of visual and audiovisual speech. Experiments 2 and 3 demonstrated that these results are dependent on intact and upright facial contexts, but only with extraoral movement displays.

The effect of varying talker identity and listening conditions on gaze behavior during audiovisual speech perception

“…Taken together, this suggests a strategy where the central and lower parts of the face are being monitored preferentially. Visual speech information is broadly distributed across the face (Yehia et al 1998), although the lower half of the face contains the lion's share of the information (Thomas & Jordan, 2004).…”

“…Taken together, this suggests a strategy where the central and lower parts of the face are being monitored preferentially. Visual speech information is broadly distributed across the face (Yehia et al 1998), although the lower half of the face contains the lion's share of the information (Thomas & Jordan, 2004).The shift to monitoring the lower part of the face suggests a subtle shift in strategy where subjects may be preferentially gathering information that provides a benefit to intelligibility. The fact that gaze behavior isn't entirely shifted to the mouth, but rather seems to be clustered around the nose suggests that subjects might be using the nose as a central vantage point that permits a monitoring the eyes and the face for social information while still moving somewhat closer to the lower part of the face.…”

Visibility of speech articulation enhances auditory phonetic convergence

“…Within the speech perception literature, extensive evidence suggests that visibility of the dynamic movements of the mouth is important for lipreading (e.g., Greenberg & Bode, 1968;IJsseldijk, 1992;Jackson, Montgomery, & Binnie, 1976) and audiovisual speech perception (e.g., Rosenblum, Johnson, & Saldaña, 1996;Thomas & Jordan, 2004). In fact, when identifying audiovisual speech presented in auditory noise, perceivers will spend more time gazing toward the mouth than toward other areas of the visible face (e.g., Paré, Richler, ten Hove, & Munhall, 2003;Vatikiotis-Bateson, Eigsti, Yano, & Munhall, 1998).…”

Similarity structure in visual speech perception and optical phonetic signals

“…Another global approach has been to present isolated face parts or faces with parts removed (e.g., Benoît, GuiardMarigny, Le Goff, & Adjoudani, 1996;IJsseldijk, 1992;Marassa & Lansing, 1995;Scheinberg, 1980;Thomas & Jordan, 2004). This research has been criticized on the grounds that face parts were not necessarily isolated adequately, that the displays encouraged unnatural attentional strategies, that configural properties of the stimuli were disrupted, and that the isolation techniques introduced potentially unnatural elements, such as edges around the relevant stimulus (Thomas & Jordan, 2004).…”

“…This research has been criticized on the grounds that face parts were not necessarily isolated adequately, that the displays encouraged unnatural attentional strategies, that configural properties of the stimuli were disrupted, and that the isolation techniques introduced potentially unnatural elements, such as edges around the relevant stimulus (Thomas & Jordan, 2004). Thomas and Jordan isolated the lip area within the face and produced stimuli without edges around the visible moving area of the face.…”