Comparison between different feature extraction techniques for audio-visual speech recognition

Abstract

Having a robust speech recognition system that can be relied upon in different environments is a strong requirement for modern systems. However audioonly speech recognition still lacks robustness when the signal to noise ratio decreases. This is especially true when the system is deployed in public spaces or is used for crises situations management where the background noise is expected to be extremely large. The video information is not affected by noise which makes it an ideal candidate for data fusion. The acoustic features have been well defined during the course of the years, the most used features being mel-frequency cepstral coefficiens (MFCCs) or linear predictive coefficients(LPCs). On the visual side, however, there is still much place for improvements. It is still not clear which visual features retain the most speech related information. Until now the visual features used were static features which describe the face of the user at one instance in time only. In the paper [1] the authors have shown that most of the techniques used for extraction of static visual features result in equivalent features or at least the most informative features exhibit this. This means that all techniques describe the same aspect of the visual stream. However the improvement of recognition even though looks promis- ing is still modest. We argue that the main problem of existing methods is that the resulting features contain no information about the motion of the speaker's lips. We present in this paper a new method for extracting useful features from the point of view of speech recognition based on optical flow analysis. The video features extracted using this method are preserving the information about speaker mouth motion. We tested the method on an audio-video database for Dutch language. The Audio-Visual Speech Recognizer(AVSR) used is based on HMMs method and was trained for large vocabulary continuous speech. For completion we also present the method introduced in the paper [2] for extracting static visual features. We will compare these two methods with respect to the induced recognition performance. Another way to recover motion information from static features is to use their first and/or the second derivative as visual features. However this can not always guaranty that the resulted features are physically sound quantities. We will also present for comparison the recognition results based on such features. The evaluation of these methods will be done under different noise conditions. We show that the audio-video recognition based on the true motion features outperforms the other settings in low Signal to Noise Ratio(SNR) conditions.

Điều Khiển Robot Pioneer P3-Dx Bằng Tiếng Nói Với Đặc Trưng MFCC Và Giải Thuật Naïve Bayes Nearest Neighbors

“…Những sản phẩm (công cụ) đã được áp dụng nhiều trong thực tế, nhận dạng và xử lý âm thanh. Hệ thống nhận dạng tiếng nói bao gồm hai bước chính [5], [6], [7]: rút trích và biểu diễn đặc trưng, huấn luyện mô hình máy học nhận dạng. Rút trích và biểu diễn đặc trưng tín hiệu âm thanh thường được sử dụng [6] Trong bài viết này, chúng tôi đề xuất hệ thống nhận dạng tiếng nói để điều khiển robot Pioneer P3-DX theo thời gian thực.…”

“…Hệ thống nhận dạng tiếng nói bao gồm hai bước chính [5], [6], [7]: rút trích và biểu diễn đặc trưng, huấn luyện mô hình máy học nhận dạng. Rút trích và biểu diễn đặc trưng tín hiệu âm thanh thường được sử dụng [6] Trong bài viết này, chúng tôi đề xuất hệ thống nhận dạng tiếng nói để điều khiển robot Pioneer P3-DX theo thời gian thực. Hệ thống thực hiện rút trích đặc trưng âm thanh MFCC, không cần qua bước tiền xử lý và biểu diễn phức tạp, hệ thống sử dụng giải thuật NBNN (Naïve Bayes Nearest Neighbor [2]) để nhận dạng trực tiếp tiếng nói là các lệnh điều khiển.…”

Automatic Visual Speech Recognition

“…LBP was developed by Timo Ojala and Matti Pietikainen and presented in (Ojala & Pietikainen, 1997). A special place in this class is taken by the feature vectors that are based on Optical Flow Analysis (OFA) (Mase & Pentland, 1991); (Martin, 1995); (Gray et al, 1997); (Fleet et al, 2000); (Iwano et al, 2001); (Tamura et al, 2002); (Furui, 2003); (Yoshinaga et al, 2003); (Yoshinaga et al, 2004); ; (Chitu et al, 2007); (Chitu & Rothkrantz, 2009). The optical flow is defined as "the apparent velocity field in an image".…”

“…For instance (Dupont & Luettin, 2000) and (Luettin et al, 1996) combine ASM with PCA features and (Chiou & Hwang, 1997) combines snake features with PCA. It was shown that the tongue, teeth and cavity have great influence on lip reading (Williams et al, 1998), therefore, the addition of these appearance related elements has significant influence on the performance of lip reading (Chitu et al, 2007). A special example is the so called Active Appearance Models (AAM) (Cootes et al, 1998) which combines the ASM method with texture based information to accurately detect the shape of the mouth or the face.…”

Speech Query Recognition for Tamil Language Using Wavelet and Wavelet Packets

“…Despite its limitations, speech recognition technology is valuable tool in many applications like live subtitling on television, dictation in Medical transcriptions, Command Control in Robotics, speech to text conversion for note making systems, replacement of keyboard and mouse for physically or visually challenged people. In noisy environment the accuracy level of ASR system will suffer greatly [1]. To recover original speech from noisy speech signal several speech de-noising methods are available.…”