Lip reading using optical flow and support vector machines

Shaikh, Ayaz A.; Kumar, Deepak; Yau, Wai Chee; Azemin, Mohd Zulfaezal Che; Gubbi, Jayavardhana

doi:10.1109/cisp.2010.5646264

Cited by 56 publications

(22 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…However, this technique was found to be sensitive to environmental conditions as information related to time was lost following the combination into a single image. In [4], a technique was described that characterized lip motion by computing its optical flow (OF), with the statistical properties of the vertical OF component being used to form the feature vectors for training a support vector machine classifier. One drawback of this approach is that OF methods are known to be sensitive to the scaling and rotation of the images under analysis.…”

Section: Introductionmentioning

confidence: 99%

Geometry based lip reading system using Multi Dimension Dynamic Time Warping

Ibrahim

Mulvaney

2012

2012 Visual Communications and Image Processing

View full text Add to dashboard Cite

This paper describes an automatic lip reading system consisting of two main modules 1) a pre-processing module able to extract lip geometry information from the video sequence and 2) a classification module to identify the visual speech based on dynamic lip movements. The recognition performance of the proposed system has been assessed in the recognition of the English digits 0 to 9 as spoken by the speakers in the video sequences available in the CUAVE database. Extraction of lip geometry features was carried out using a combination of a skin color filter, a border following algorithm and a convex hull approach. The proposed method was compared with the popular 'snake' technique and was found to improve lip shape extraction performance for the database studied. Lip geometry features including height, width, ratio, area, perimeter and various combinations of these features were evaluated to determine which performs the best when representing speech in the visual domain in the application of three separate classification methods, namely optical flow, Dynamic Time Warping (DTW) and a new approach termed Multi-Dimensional DTW. Experiments show that the proposed system is capable of a recognition performance of 68% just using lip height, lip width and the ratio of these features demonstrating that the system has the potential to be incorporated in a multimodal speech recognition system for use in noisy environments.

show abstract

Section: Introductionmentioning

confidence: 99%

Geometry based lip reading system using Multi Dimension Dynamic Time Warping

Ibrahim

Mulvaney

2012

2012 Visual Communications and Image Processing

View full text Add to dashboard Cite

show abstract

“…They used data base of hearing impaired person and observed that DWT with HMM gives better result. A. Shaikh et al [9] used optical flow information as a feature vector for lip reading. The vocabulary used in their experiment was viseme.…”

Section: Introductionmentioning

confidence: 99%

Visual Lip Reading using 3D-DCT and 3D-DWT and LSDA

Morade¹,

Patnaik²

2016

IJCA

View full text Add to dashboard Cite

Human uses visual information while trying to understand speech, especially in noisy conditions or in situations where the audio signal is not available. Lip reading is the technique of a comprehensive understanding the underlying speech by processing on the movement of lips. However, the recognition of lip motion is a difficult task since the region of interest (ROI) is nonlinear and noisy. In proposed method lip reading system we have used two stage feature extraction model which is precised, discriminative and computation efficient. The first stage 3D Discrete Wavelet Transform (3D-DWT) or 3D Discrete Cosine Transform (3D-DCT) is used and the second stage is Locality Sensitive Discriminant Analysis (LSDA) to trim down the feature dimensions. These features make a novel lip reading system with small feature vector size. In addition to the novel feature extraction technique, the performance of Naive Bayes and SVM classifier is compared. CUAVE database of 0 to 9 utterances in English is used for experimentation. Results of 3 dimension transform with LSDA are compared with 2 dimension transform with LSDA. Experimental results show that 3D-DWT+LSDA feature mining are compared with 3D-DWT with PCA or LDA. 3D-DWT+LSDA result is also compared with 3D-DCT + LSDA.

show abstract

“…To assess the performance of the new shape-based approach with respect to motion-based and appearance-based techniques, both OF and DCT methods were implemented in the manner described in [24] and [66]. The same lip images used to extract the lip geometry were supplied to the OF and DCT implementations, producing respectively recognition rates of 26.94% and 57.92%.…”

Section: Classificationmentioning

confidence: 99%

“…In addition, information about the timing of movements was lost following the combination of sequences into a single image, resulting in lower performance. In [24], the authors reported a technique that computed the optical flow (OF) of lip motions in a video data stream. The statistical properties of the vertical OF component were used to form feature vectors suitable for training a support vector machine (SVM) classifier.…”

Section: Introductionmentioning

confidence: 99%