Multimodal Grounding for Sequence-to-sequence Speech Recognition

Abstract

Humans are capable of processing speech by making use of multiple sensory modalities. For example, the environment where a conversation takes place generally provides semantic and/or acoustic context that helps us to resolve ambiguities or to recall named entities. Motivated by this, there have been many works studying the integration of visual information into the speech recognition pipeline. Specifically, in our previous work, we propose a multistep visual adaptive training approach which improves the accuracy of an audio-based Automatic Speech Recognition (ASR) system. This approach, however, is not end-to-end as it requires fine-tuning the whole model with an adaptation layer. In this paper, we propose novel end-to-end multimodal ASR systems and compare them to the adaptive approach by using a range of visual representations obtained from state-of-the-art convolutional neural networks. We show that adaptive training is effective for S2S models leading to an absolute improvement of 1.4% in word error rate. As for the end-to-end systems, although they perform better than baseline, the improvements are slightly less than adaptive training, 0.8 absolute WER reduction in singlebest models. Using ensemble decoding, end-to-end models reach a WER of 15% which is the lowest score among all systems.

Multimodal and Multiresolution Speech Recognition with Transformers

“…Our AV-ASR system yields gains > 3% over audio only models for both subword and multiresolution predictions. Finally, we observe that while the Listen, Attend and Spell-based architecture of (Caglayan et al, 2019) is consistent across models. It is important to note that our models are trained end-to-end with both audio and video features.…”

“…For example, a basketball court is more likely to include the term "lay-up" whereas an office place is more likely include the term "layoff". This approach can boost ASR performance, while the requirements for video input are kept relaxed (Caglayan et al, 2019;Hsu et al, 2019). Additionally we consider a multiresolution loss that takes into account transcriptions at the character and subword level.…”

“…Multimodal grounding for ASR systems has been explored in (Caglayan et al, 2019), where a pretrained RNN-based ASR model is finetuned with visual information through Visual Adaptive Training. Sterpu et al (2018) propose a seq2seq model based on RNNs for lip-reading that performs cross-modal alignment of face tracking and audio features through an attention mechanism.…”

Looking Enhances Listening: Recovering Missing Speech Using Images

“…However, previous work has suggested that visual modality is only being used as a regularization signal [5]. To investigate this behavior, we also measure the models' ability to recover masked words -ensuring that the observed improvements come from the semantics of the visual context.…”

“…Previous works show improvements in the domains of visual question-answering [1], multimodal machine translation [2], visual dialog [3], and automatic speech recognition (ASR) [4]. Although there are several visual adaption approaches for ASR [4,5,6,7,8,9], it is still unclear how the models leverage the visual modality.…”