"Notic My Speech" -- Blending Speech Patterns With Multimedia

Abstract

Speech as a natural signal is composed of three parts -visemes (visual part of speech), phonemes (spoken part of speech), and language (the imposed structure). However, video as a medium for the delivery of speech and a multimedia construct has mostly ignored the cognitive aspects of speech delivery. For example, video applications like transcoding and compression have till now ignored the fact how speech is delivered and heard. To close the gap between speech understanding and multimedia video applications, in this paper, we show the initial experiments by modelling the perception on visual speech and showing its use case on video compression. On the other hand, in the visual speech recognition domain, existing studies have mostly modeled it as a classification problem, while ignoring the correlations between views, phonemes, visemes, and speech perception. This results in solutions which are further away from how human perception works. To bridge this gap, we propose a view-temporal attention mechanism to model both the view dependence and the visemic importance in speech recognition and understanding. We conduct experiments on three public visual speech recognition datasets. The experimental results show that our proposed method outperformed the existing work by 4.99% in terms of the viseme error rate. Moreover, we show that there is a strong correlation between our model's understanding of multi-view speech and the human perception. This characteristic benefits downstream applications such as video compression and streaming where a significant number of less important frames can be compressed or eliminated while being able to maximally preserve human speech understanding with good user experience.

Multi-Perspective LSTM for Joint Visual Representation Learning

“…iii) We also consider other alternative recognition solutions available in the literature for each experiment. For lip reading, we include PCA+LSTM+HMM [33], CNN+LSTM [16], 4-layer CNN+Hierarchical LSTM [22], and VGG-M+Attentive Bi-LSTM [26], as they use LSTM networks in combination with different spatial feature extractors and classifiers. 3DCNN [16] has also been considered as it has shown to be an effective alternative to CNN + LSTM architectures.…”

“…To this end, we use the same 4-layer CNN used in [22], thus solely comparing our MP-LSTM network with the hierarchical LSTM network proposed in [22]. The results show the superiority of our MP-LSTM when compared to attentive Bi-LSTM [26] and hierarchical LSTM [22]. Experiment 2 (LF-based Face Recognition): Table 4 presents the face recognition performance when, respectively: i) LFFC is used for training and LFFW for testing (Protocol 1); and ii) LFFW is used for training and LFFC for testing (Protocol 2).…”