Probing the Need for Visual Context in Multimodal Machine Translation

Abstract

Current work on multimodal machine translation (MMT) has suggested that the visual modality is either unnecessary or only marginally beneficial. We posit that this is a consequence of the very simple, short and repetitive sentences used in the only available dataset for the task (Multi30K), rendering the source text sufficient as context. In the general case, however, we believe that it is possible to combine visual and textual information in order to ground translations. In this paper we probe the contribution of the visual modality to state-of-the-art MMT models by conducting a systematic analysis where we partially deprive the models from source-side textual context. Our results show that under limited textual context, models are capable of leveraging the visual input to generate better translations. This contradicts the current belief that MMT models disregard the visual modality because of either the quality of the image features or the way they are integrated into the model.

Distilling Translations with Visual Awareness

“…They perform careful experiments by using input degradation and observe that, specially under limited textual context, multimodal models exploit the visual input to generate better translations. Caglayan et al (2019) also show that MMT systems exploit visual cues and obtain correct translations even with typographical errors in the source sentences. In this paper, we build upon this idea and investigate the potential of visual cues for refining translation.…”

“…We note that translation refinement is different translation re-ranking from a text-only model based on image representation (Shah et al, 2016;Hitschler et al, 2016;, since the latter assumes that the correct translation can already be produced by a text-only model. Caglayan et al (2019) investigate the importance and the contribution of multimodality for MMT. They perform careful experiments by using input degradation and observe that, specially under limited textual context, multimodal models exploit the visual input to generate better translations.…”

Looking Enhances Listening: Recovering Missing Speech Using Images

“…This observation indicates that the model is using the visual modality as a regularization technique, and not using the semantics of the image. Similar findings in multimodal machine translation [11] led to an investigation of the utility of visual context [12]. They concluded that visual modality is helpful when the primary modality is degraded.…”

“…1. Silence Masking: We substitute the masked word with a specific value, silence, similar to the special token in [12]. In this nonrealistic scenario, the model is trained to generate the missing word when a known signal is present in the audio.…”

“…In this paper, inspired by [12], we hypothesize that Multimodal ASR systems can take advantage of the visual modality under severely noisy conditions and be more robust than unimodal models. Such robustness to noise is particularly desirable in ASR -since speech is inherently a noisy medium.…”