Continuous audio-visual speech recognition

Abstract

Abstract. We address the problem of robust lip tracking, visual speech feature extraction, and sensor integration for audio-visual speech recognition applications. An appearance based model of the articulators, which represents linguistically important features, is learned from example images and is used to locate, track, and recover visual speech information. We t a c kle the problem of joint temporal modelling of the acoustic and visual speech signals by applying Multi-Stream hidden Markov models. This approach allows the use of di erent temporal topologies and levels of stream integration and hence enables to model temporal dependencies more accurately. T h e system has been evaluated for a continuously spoken digit recognition task of 37 subjects. IDIAP{RR 98-02

Comparison between different feature extraction techniques for audio-visual speech recognition

“…Other models were developed by placing constraints on the states or the transitions in order to make the new models tractable. The Multi-Stream HMM [41,42] allows for multiple input feature streams that may have different frame rates and can be asynchronous. It assumes that the model consists of a number of sub-unit models that correspond to the level at which the streams have to synchronize, for example phoneme level or syllable level.…”

The fusion of visual lip movements and mixed speech signals for robust speech separation

“…The goal is to use the motion of the lips in order to improve the acoustic recognition of the words. Many different studies have shown improved speech recognition (both faster and more accurate) when visual cues are available [6][7][8][9][10][11][16][17][18][19]21,22].…”

“…In the first stage, information from the video frames is processed in order to prepare it for integration with the acoustic signal [7]. One simplistic example of this is image-based data extraction, during which the image of the mouth is selected without any processing [7,19,20,22]. While all the information contained within that frame is automatically selected, it does not include any dimensionality reduction and hence makes audiovisual information fusion extremely difficult.…”

“…Other first stage techniques include visual motion based information extraction approaches [7,23], model based approaches [5,7,[13][14][15], and geometric feature extraction techniques [7,18]. The latter techniques involve the automatic detection of pertinent features (such as the width and height of the outer corners of the mouth) that are then used for audiovisual information fusion.…”

“…Audiovisual information fusion can either be accomplished early in the speech recognition process [20,21] or late [18,19]. Early integration is often preferable since it will result in a more robust system [7]. Typical systems in the Ôearly integration' category often combine audiovisual cues before the phonetic classification stage.…”