The Dynamics of Audiovisual Behavior in Speech

Abstract

While it is well-known that faces provide linguistically relevant information during communication, most efforts to identify the visual correlates of the acoustic signal have focused on the shape, position and luminance of the oral aperture. In this work, we extend the analysis to full facial motion under the assumption that the process of producing speech acoustics generates linguistically salient visual information, which is distributed over large portions of the face. Support for this is drawn from our recent studies of the eye movements of perceivers during a variety of audiovisual speech perception tasks. These studies suggest that perceivers detect visual information at low spatial frequencies and that such information may not be restricted to the region of the oral aperture. Since the biomechanical linkage between the facial and vocal tract systems is one of close proximity and shared physiology, we propose that physiological models of speech and facial motion be integrated into one audiovisual model of speech production. In addition to providing a coherent account of audiovisual motor control, the proposed model could become a useful experimental tool, providing synthetic audiovisual stimuli with realistic control parameters.3

Perceptual Evaluation of Video-Realistic Speech

“…In particular, the authors made a strong argument for phonetic perception (see also Berenstein, 2003) rather than viseme clusters (as in Cohen et al1996 andOwens andBlazek 1985). This point is also strongly supported by the notion that "the moving vocal tract simultaneously shapes the acoustics and the motion of the face" (Vatikiotis-Bateson et al 1996). ---Thus there is the necessity for scrutiny of visual phonetic articulation.…”

Seeing speech in space and time: psychological and neurological findings

“…Both mouthshape (and the visibility of mouth parts) and mouth movement (the dynamics of mouth actions, including rate of speech) play their part. This insight is confirmed by a range of experimental findings 18,20,22 and by findings in applied telematics which show that speechreading accuracy for audiovisual inputs with auditory dynamic noise falls off as framerate (temporal resolution) of the display of the speaker's face drops from about 30Hz to [8][9][10][11][12]24 In addition, seen rate of speech can be readily discriminated and can directly affect the identification of a heard speech token 5,6 But how does this work? Is one process subsumed in the other?…”

The Paradigm Shift to Multimodality in Contemporary Computer Interfaces

“…Historically, multimodal speech and lip movement research was driven by cognitive science interest in intersensory audio-visual perception and the coordination of speech output with lip and facial movements (Benoit and Le Goff, 1998;Bernstein and Benoit, 1996;Cohen and Massaro, 1993;Massaro and Stork, 1998;McGrath and Summerfield, 1985;McGurk and MacDonald, 1976;McLeod and Summerfield, 1987;Robert-Ribes et al, 1998;Sumby and Pollack, 1954;Summerfield, 1992;Vatikiotis-Bateson et al, 1996). Among the many contributions of this literature was a detailed classification of human lip movements (visemes) and the viseme-phoneme mappings that occur during articulated speech.…”

“…The cognitive science literature has provided information on the integration patterns that typify people's speech, lip, and facial movements Ekman, 1992;Ekman and Friesen, 1978;Fridlund, 1994;Hadar et al, 1983;Massaro and Cohen, 1990;Stork and Hennecke, 1995;Vatikiotis-Bateson et al, 1996), speech with pen input, and speech with manual gestures (Kendon, 35 1980;McNeill, 1992;Oviatt et al, 1997). Early work on multimodal systems focused exclusively on processing point-and-speak integration patterns during selection actions.…”