An image transform approach for HMM based automatic lipreading

Abstract

This paper concentrates o n the visual front end for hidden Markov model based automatic lipreading. Two approaches for extracting features relevant to lipreading, given image sequences of the speaker's mouth region, are considered: A lip contour based feature approach, which first obtains estimates of the speaker's lip contours and subsequently extracts features from them, and an image transf o r m based approach, which obtains a compressed representation of the image pixel values that contain the speaker's mouth. Various possible features are considered i n each approach, and experimental results o n a number of visualonly recognition tasks are reported. It is shown that the image transform based approach results i n superior lipreading performance. In addition, feature mean subtraction is demonstrated to improve performance i n multi-speaker and speaker-independent recognition tasks. Finally, the effects of video degradations to image transform based automatic lipreading are studied. It is shown that lipreading performance dramatically deteriorates below a 10 Hz field rate, and that image transform features are robust to noise and compression artifacts.

Discrimination Between Native and Non-Native Speech Using Visual Features Only

“…We investigate five different appearance-based descriptors: Principal Component Analysis (PCA) [36], 2D Discrete Cosine Transform (DCT) [36], Discrete Wavelet Transform (DWT) [36], Local Binary Patterns (LBP) [37] and Histograms of Oriented Gradients (HOG) [38], all calculated on pixel intensities. We choose to investigate these appearance descriptors because they have proven to be highly descriptive and informative of facial expression changes by numerous studies on automatic facial expression recognition (e.g., [39]- [41]).…”

“…These appearance features include PCA, DCT, DWT, LBP and HOG. The image-transform-based descriptors, i.e., PCA, DCT and DWT [36], are the most commonly used feature representations for visual speech processing tasks [50], [51]. LBP has been widely used as a robust image compression technique for texture representation [37], and it is one of the most commonly used facial appearance descriptors in face recognition and facial expression recognition [39], [40].…”

Geometrical-based lip-reading using template probabilistic multi-dimension dynamic time warping

“…As the application domain is the same, lip reading classification techniques are often the same as those applied in the audio speech recognition (ASR) field and, consequently, dynamic time warping (DTW) [39,40] and HMMs [10,18,41], are popular. Moreover, by using a method common to both the audio and visual aspects of speech, there is the potential for a more straightforward combination of results obtained from separate audio and visual investigations and such integration has often been carried out using machine learning techniques, such as time delay neural network (TDNN) [42], support vector machines (SVM) [43] and AdaBoost [44].…”

“…As the area covered by such a mouth region can contain a large number of pixels (for example, assuming each color is represented in 8 bits, a 128 Â 128 pixel region in RGB space will have a total of 49,152 pixels), a transformation to fewer dimensions is needed to make appearance-based approaches computationally manageable. Such transformations are typically borrowed from the image compression and pattern classification literature, such as principal components analysis (PCA) [11], the discrete cosine transform (DCT) [9], the discrete wavelet transform (DWT) [10] and linear discriminant analysis (LDA) [34].…”

“…In [8], the results of visual ASR experiments involving the use of the IBM ViaVoice database were presented in their comparison of four types of visual features, namely discrete cosine transform (DCT) [9], discrete wavelet transform (DWT) [10], principal components analysis (PCA) [11], and active appearance models (AAM) [12]. A solution using hidden Markov models (HMMs) [13] as the classifier found that DCT based visual features were the most promising for the recognition task.…”