Visible Speech as a Function of Image Quality: Effects of Display Parameters on Lipreading Ability

Abstract

Two experiments are reported that examine effects of certain visual display parameters on subjects' ability to lipread numbers. The quality of the image of the speaker's face is altered by varying temporal resolution (frame rate), spatial resolution (number of pixels) and contrast resolution (number of grey scale levels). In Experiment 1, combinations of these variables are examined in a repeated measures design and reliable effects of reducing both frame rate and number of grey levels are evident. In Experiment 2, the frame rate and grey scale settings are factorially combined on a between‐subjects basis, and practice is given so that subjects can adapt to the display conditions to which they are exposed. Effects of frame rate and grey scale are again statistically significant, and additive. Lipreading performance improves over the short duration of the experiment, but both image variables continue to affect performance. The effect size of the grey scale factor decreases with practice, but not that of frame rate. The results are discussed with regard to the kinds of relevant visual cues that may be affected by image degradation. Implications for the planned development of low cost digital communication are also considered. Trading off one image variable at the expense of another may be necessary to meet bandwidth limitations, and data such as those presented will be necessary to decide on a combination of parameter values to satisfy human performance and technical criteria.

The Natural Statistics of Audiovisual Speech

“…This range of temporal modulations is important for audiovisual speech perception. For example, reduction of the frame rate of visual speech removes the audiovisual benefit [7] , [60] , [61] . Similarly, audiovisual advantages disappear for synthetic talkers speaking at rates faster than normal speech [40] .…”

Modulation transfer functions for audiovisual speech

“…Degrading visual temporal cues, e.g. by reducing the frame rate in videos of the speaker’s face, reduces the AV perception benefit [ 92 , 93 ]. Non-oral facial movements also contribute to AV perception as seen by the fact that AV perception benefits occur when the mouth is visually occluded [ 94 ].…”

Temporal filtering of visual speech for audio-visual speech recognition in acoustically and visually challenging environments

“…Thus, the noise effect can be effectively reduced by removing the high frequency components as long as the components do not contain important speech information. In psychological studies, it has been experimentally shown that the frame rate of 16.7 Hz is sufficient for human visual speech perception and a higher frame rate is not much helpful [27], which implies important speech information lies below 8.35 Hz. Thus, the higher cut-off frequency of the BPF is set to 9 Hz based on the spectral and the psychological analyses.…”