Lip reading is the ability to partially understand speech by looking at the speaker's lips. It improves the intelligibility of speech in noise when audio-visual perception is compared with audio-only perception. A recent set of experiments showed that seeing the speaker's lips also enhances sensitivity to acoustic information, decreasing the auditory detection threshold of speech embedded in noise [J. Acoust. Soc. Am. 109 (2001) 2272; J. Acoust. Soc. Am. 108 (2000) 1197]. However, detection is different from comprehension, and it remains to be seen whether improved sensitivity also results in an intelligibility gain in audio-visual speech perception. In this work, we use an original paradigm to show that seeing the speaker's lips enables the listener to hear better and hence to understand better. The audio-visual stimuli used here could not be differentiated by lip reading per se since they contained exactly the same lip gesture matched with different compatible speech sounds. Nevertheless, the noise-masked stimuli were more intelligible in the audio-visual condition than in the audio-only condition due to the contribution of visual information to the extraction of acoustic cues. Replacing the lip gesture by a non-speech visual input with exactly the same time course, providing the same temporal cues for extraction, removed the intelligibility benefit. This early contribution to audio-visual speech identification is discussed in relationships with recent neurophysiological data on audio-visual perception.

A labial perturbation of the French rounded vowel [u] was used to examine the respective weights of the articulatory and acoustic levels in the control of vowel production. A 20-mm-diam lip tube was inserted between the lips of the speakers. Acoustic and x-ray articulatory data were obtained for isolated vowel productions by 11 native French speakers in normal and lip-tube conditions. Compensation abilities were evaluated through accuracy of the F1-F2 pattern. Possible compensations were examined from nomograms using the new model of Fant [ISCLP 92 Proceedings (University of Alberta, Edmonton, 1992)]. Acoustic interpretations of the articulatory changes were made by generating area functions from midsagittal views, used together with a harmonic acoustic model. For the first perturbed trial, immediately after the insertion of the tube, no speaker was able to produce a complete compensation, but clear differences between speakers were observed: Seven of them moved the tongue and hence limited the deterioration of the F1-F2 pattern, whereas the remaining four did not show any pertinent articulatory change. These data support the idea of speaker-specific internal representations of the articulatory-to-acoustic relationships. The results for the following 19 perturbed trials indicate that speakers used the acoustic signal in order to elaborate an optimal compensation strategy. One speaker achieved complete compensation by changing his constriction location from a velo-palatal to a velo-pharyngeal region of the vocal tract. Six others moved their tongues in the right direction, achieving partial acoustic compensation, while the remaining four did not compensate. The control of speech production thus seems to be directed toward achieving an auditory goal, but completely achieving the goal may be impossible because of speaker-dependent articulatory constraints. It is suggested that these constraints are due more to speaker-specific internal representation of articulatory-to-acoustic relationships rather than to anatomical or neurophysiological limitations. Speech control could thus be ensured partly with the use of this internal representation, and partly--particularly under perturbed conditions--by monitoring the acoustic signal. ¸ 1995 Acoustical Society of America.

No abstract