Improving Speech Recognition Performance in Noisy Environments by Enhancing Lip Reading Accuracy

Abstract

The current accuracy of speech recognition has been able to reach over 97% on different data sets, but the accuracy of speech recognition in noisy environments is greatly reduced. Improving speech recognition performance in noisy environments is a challenging task. Due to the fact that visual information is not affected by noise, researchers often use lip information to help improve speech recognition performance. This is where the performance of lip reading and the effect of cross-modal fusion are particularly important. In this paper, we try to improve the accuracy of speech recognition in noisy environments by improving the lip reading performance and the cross-modal fusion effect. First, due to the same lip may contain multiple meanings, we construct a one-to-many mapping relationship model between lips and speech, allowing the lip-reading model to consider the feasibility of which articulations are represented from the input lip movements. Also, audio representations are preserved by modeling the inter-relationships between paired audio-visual representations. At the inference stage, the preserved audio representations can be extracted from memory by the learned interrelationships using only video input. Second, a joint cross-fusion model using the attention mechanism can effectively exploit complementary inter-modal relationships, and the model calculates cross-attention weights based on the correlations between joint feature representations and individual modalities. Finally, our proposed model has a 4.0% reduction in WER in −15 dB SNR environment compared to the baseline method, and a 10.1% reduction in WER compared to speech recognition. The experimental results show that our method has a significant improvement over speech recognition models in different noise environments.

Assessing Visual Speech Communication Abilities: Cross-situational Consistency in Lipreadability

“…For example, individual differences in co-articulation may underlie difficulties in the transfer of training of visual speech recognition (Bear & Harvey, 2017). Future interactions between psychologists and computer scientists studying multimodal speech recognition could facilitate identification of specific targets for training people and/or computers, perhaps even leading to a new generation of 'smart' hearing aids that use lipreading to enhance automatic speech recognition in noisy environments (e.g., Li et al, 2023).…”

Biglip: A Pipeline for Building Data Sets for Lip-Reading

“… Forensics: lip-reading can be used to reconstruct the dialogues in a footage where the audio has been lost or it is noisy.  Automated Speech Recognition [1]: automakers can integrate lip-reading systems to complement their ASR model in order to understand commands (for example "turn on the A/C") from the driver or the passengers in smart cars when the music's volume is too high. Lip-reading is also necessary in this case to recognize the active speaker in the scene.…”

Attention-Based Fusion of Ultrashort Voice Utterances and Depth Videos for Multimodal Person Identification

“…Multimodal deep learning has emerged as a powerful approach for various tasks by combining information from different modalities, exploiting their complementary nature, and enhancing their overall performance [1][2][3][4]. In the realm of speaker recognition, incorporating multiple features, such as lip movements, depth images, and voice, can lead to improved accuracy and robustness in applications such as security systems, access control, and surveillance [2,[5][6][7][8][9].…”