Multimodal Neural Machine Translation Using CNN and Transformer Encoder

Abstract

Multimodal machine translation uses images related to source language sentences as inputs to improve translation quality. Pre-existing multimodal neural machine translation (NMT) models that incorporate the visual features of each image region into an encoder for source language sentences or an attention mechanism between an encoder and a decoder cannot catch the relationship between the visual features from each image region. This paper proposes a new multimodal NMT model that encodes an input image using a convolutional neural network (CNN) and a Transformer encoder. In particular, our proposed image encoder extracts visual features from each image region using a CNN then encodes an input image based on the extracted visual features using a Transformer encoder, where the relationship between the visual features from each image region is captured by a self-attention mechanism of the Transformer encoder. Our experiments with English-German translation tasks using the Multi30K data set showed our proposed model improves 0.96 BLEU points against a baseline Transformer NMT model without image inputs and improves 0.47 BLEU points against a baseline multimodal Transformer NMT model without a Transformer encoder for images.

Supervised Visual Attention for Multimodal Neural Machine Translation

“…8 Grönroos et al 2018;Ive et al 2019;Zhang et al 2020) (Delbrouck and Dupont 2017;Zhang et al 2020) (Calixto et al 2017;Helcl et al 2018;Libovický et al 2018;Ive et al 2019;Takushima et al 2019)…”

Supervised Visual Attention for Multimodal Neural Machine Translation

“…Most MNMT models have incorporated an input image's features with a visual attention mechanism. Some studies have introduced a visual attention mechanism that captures relationships between source language words and image regions (Delbrouck and Dupont, 2017;Zhang et al, 2020), while others have used a visual attention mechanism that captures relationships between target language words and image regions (Calixto et al, 2017;Ive et al, 2019;Takushima et al, 2019). Note that these visual attention mechanisms were trained in an unsupervised manner, and, as far as we know, a supervised visual attention mechanism has not yet been proposed.…”