End-to-end Multimodal Speech Recognition

Palaskar, Shruti; Sanabria, Ramon; Metze, Florian

doi:10.1109/icassp.2018.8462439

Cited by 31 publications

(30 citation statements)

References 35 publications

(54 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…when the video consistently provides object, action and scene level cues correlated with the speech content as may be the case with instructional videos. Here, visual cues from the recording environment (indoor vs outdoor) or the interaction between salient objects (people, instruments, vehicles, tools and equipments) can be exploited by the recognizer in various ways to learn a better acoustic and/or language model [3,4,5]. Figure 1 shows such an example where an ASR system without access to visual modality can produce an homophonic utterance like eucalylie instead of the rarely occurring correct word ukulele.…”

Section: Introductionmentioning

confidence: 99%

“…In this paper, we first apply an adaptive training scheme [3,4,5] for sequence-to-sequence (S2S) speech recognition and then propose two novel multimodal grounding methods for S2S ASR inspired from previous work in image captioning [6] and multimodal neural machine translation (MMT) [7,8]. We compare both approaches through the use of visual features extracted from pre-trained models trained for object, scene and action recognition tasks [9,10,11].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Multimodal Grounding for Sequence-to-sequence Speech Recognition

Çağlayan

Sanabria

Palaskar

et al. 2019

ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

Humans are capable of processing speech by making use of multiple sensory modalities. For example, the environment where a conversation takes place generally provides semantic and/or acoustic context that helps us to resolve ambiguities or to recall named entities. Motivated by this, there have been many works studying the integration of visual information into the speech recognition pipeline. Specifically, in our previous work, we propose a multistep visual adaptive training approach which improves the accuracy of an audio-based Automatic Speech Recognition (ASR) system. This approach, however, is not end-to-end as it requires fine-tuning the whole model with an adaptation layer. In this paper, we propose novel end-to-end multimodal ASR systems and compare them to the adaptive approach by using a range of visual representations obtained from state-of-the-art convolutional neural networks. We show that adaptive training is effective for S2S models leading to an absolute improvement of 1.4% in word error rate. As for the end-to-end systems, although they perform better than baseline, the improvements are slightly less than adaptive training, 0.8 absolute WER reduction in singlebest models. Using ensemble decoding, end-to-end models reach a WER of 15% which is the lowest score among all systems.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Multimodal Grounding for Sequence-to-sequence Speech Recognition

Çağlayan

Sanabria

Palaskar

et al. 2019

ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Second, it is possible that the ResNet posteriors are either extremely noisy, or simply fail to identify certain relevant objects because of a domain mismatch, as discussed in Section 3.1. Previous work in the context of ASR shows that using the penultimate instead of the last layer of the ResNet makes little difference [9].…”

Section: Discussionmentioning

confidence: 99%

“…In audio-visual speech recognition, [9,10] explore strategies to learn and fuse audio and visual representations in a neural net, including concatenating both modalities, bilinear products between representations, and weighted addition of modalities. Within a scene classification task, [11] use one neural network per modality.…”

Section: Related Workmentioning

confidence: 99%

Learning from Multiview Correlations in Open-domain Videos

Holzenberger

Palaskar

Madhyastha

et al. 2019

ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

An increasing number of datasets contain multiple views, such as video, sound and automatic captions. A basic challenge in representation learning is how to leverage multiple views to learn better representations. This is further complicated by the existence of a latent alignment between views, such as between speech and its transcription, and by the multitude of choices for the learning objective. We explore an advanced, correlation-based representation learning method on a 4-way parallel, multimodal dataset, and assess the quality of the learned representations on retrieval-based tasks. We show that the proposed approach produces rich representations that capture most of the information shared across views. Our best models for speech and textual modalities achieve retrieval rates from 70.7% to 96.9% on open-domain, user-generated instructional videos. This shows it is possible to learn reliable representations across disparate, unaligned and noisy modalities, and encourages using the proposed approach on larger datasets.Index Terms-Multiview Learning, Representation Learning, Canonical Correlation Analysis * Equal contribution in its Portuguese translation. Figure 1 shows an overview of our learning algorithm with 4 different input views, detailed in Section 3.Given these multiple parallel modalities, we address the following: how much information is shared across modalities? How can we learn a representation that captures information from all modalities? We measure this using intrinsic evaluations. In the remainder of this paper, we describe related work, then the proposed methods and our experimental setup, and finally present our results. CCA loss neural network English speech neural network video neural network Once I have my jack stand there on the rear axle, go ahead and release the hydraulic pressure.. English text neural network Q u a n d o e u t i v e r m e u macaco parado no eixo traseiro, vá em frente e libere a pressão hidráulica.

show abstract

“…Consequently, visual modality integration has recently become a trend in the speech and natural language processing communities. Previous works show improvements in the domains of visual question-answering [1], multimodal machine translation [2], visual dialog [3], and automatic speech recognition (ASR) [4]. Although there are several visual adaption approaches for ASR [4,5,6,7,8,9], it is still unclear how the models leverage the visual modality.…”

Section: Introductionmentioning

confidence: 99%

Looking Enhances Listening: Recovering Missing Speech Using Images

Srinivasan

Sanabria

Metze

2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

Speech is understood better by using visual context; for this reason, there have been many attempts to use images to adapt automatic speech recognition (ASR) systems. Current work, however, has shown that visually adapted ASR models only use images as a regularization signal, while completely ignoring their semantic content. In this paper, we present a set of experiments where we show the utility of the visual modality under noisy conditions. Our results show that multimodal ASR models can recover words which are masked in the input acoustic signal, by grounding its transcriptions using the visual representations. We observe that integrating visual context can result in up to 35% relative improvement in masked word recovery. These results demonstrate that end-to-end multimodal ASR systems can become more robust to noise by leveraging the visual context.

show abstract

End-to-end Multimodal Speech Recognition

Cited by 31 publications

References 35 publications

Multimodal Grounding for Sequence-to-sequence Speech Recognition

Multimodal Grounding for Sequence-to-sequence Speech Recognition

Learning from Multiview Correlations in Open-domain Videos

Looking Enhances Listening: Recovering Missing Speech Using Images

Contact Info

Product

Resources

About