Noise-Robust Speech Recognition System based on Multimodal Audio-Visual Approach Using Different Deep Learning Classification Techniques

Abstract

Multimodal speech recognition is proved to be one of the most promising solutions for designing a robust speech recognition system, especially when the audio signal is affected by noise. The visual signal can be used to obtain more information to enhance the recognition accuracy in a noisy system, whereas the reliability of the visual signal is not affected by the acoustic noise. The critical stage in designing a robust speech recognition system is the choice of an appropriate feature extraction method for both audio and visual signals and the choice of a reliable classification method from a large variety of existing classification techniques. This paper extends an earlier work on designing a speech recognition system based on Hidden Markov Model (HMM) classification technique of using visual modality in addition to audio modality[1]. Improved off traditional HMM-based Automatic Speech Recognition (ASR) accuracy is achieved by implementing a technique using either RNN-based or CNN-based approach. This research is intending to deliver two contributions: The first contribution is the methodology of choosing the visual features by comparing different visual features extraction methods like Discrete Cosine Transform (DCT), blocked DCT, and Histograms of Oriented Gradients with Local Binary Patterns (HOG+LBP), and applying different dimension reduction techniques like Principal Component Analysis (PCA), auto-encoder, Linear Discriminant Analysis (LDA), t-distributed Stochastic Neighbor Embedding (t-SNE) to find the most effective features vector size. Then the obtained visual features are early integrated with the audio features obtained by using Mel Frequency Cepstral Coefficients (MFCCs) and the combined audiovisual feature vector is fed to the classification process. The second contribution of this research is the methodology of developing the classification process using deep learning by comparing different Deep Neural Network (DNN) architectures like Bidirectional Long-Short Term Memory (BiLSTM) and Convolution Neural Network (CNN) with the traditional HMM. The proposed model is evaluated on two multispeakers AV-ASR datasets named AVletters 1 and GRID 2 with different SNRs. The model performs speaker-independent experiments in AVlettter dataset and speaker-dependent in GRID dataset. The experimental results obtained in this research showed that using early integration between audio features obtained by MFCC and visual features obtained by DCT with zigzag scanning demonstrate higher recognition accuracy when used with BiLSTM classifier compared to other methods for features extraction, dimension reduction, and classification techniques.

Arabic Automatic Speech Recognition Based on Emotion Detection

“…PCA uses statistical tools to identify noise and redundancy in the dataset [30]. It keeps the necessary parts that have more variation of the data and removes the unnecessary parts with fewer variations, therefore speeding up the training and testing time of the machine learning algorithm.…”

Convolutional Neural Network for Arabic Speech Recognition

“…Alternatively, it is known that the CNN model is a deep learning algorithm that can perform complex tasks with images, videos, texts, and sounds that are inspired by the human visual system [3]. CNN's achieved great success in image recognition [4], and recently they are widely adopted in ASR systems [5]- [9]. Most leading technology companies like Google, Facebook, Microsoft, IBM, Yahoo!, Twitter and Adobe, have initiated research and development projects [10]- [13] which employs CNN for image recognition products and services.…”