Using Visual Feature Space as a Pivot Across Languages

Abstract

Our work aims to leverage visual feature space to pass information across languages. We show that models trained to generate textual captions in more than one language conditioned on an input image can leverage their jointly trained feature space during inference to pivot across languages. We particularly demonstrate improved quality on a caption generated from an input image, by leveraging a caption in a second language. More importantly, we demonstrate that even without conditioning on any visual input, the model demonstrates to have learned implicitly to perform to some extent machine translation from one language to another in their shared visual feature space. We show results in German-English, and Japanese-English language pairs that pave the way for using the visual world to learn a common representation for language.

Backpropagation-Based Decoding for Multimodal Machine Translation

“…The variable γ is initialized as γ = f 1 ( ϵ ) where ϵ is a dummy input image where every pixel value is sampled from a uniform distribution . This extends our earlier reported method ( Yang et al, 2020 ) and demonstrates the validity of this type of approach in the trilingual scenario where one set of language pairs is not represented in the training annotations. We use a held-out set to determine the optimal amount of updates in the optimization process in Eq.…”

“…For instance, multimodal machine translation is most effective when images are provided on top of parallel text where the images enhance the traditional machine translation corpora, and a second limitation is that translation models are still required for every language pair even if there is a single common visual representation. The present work significantly extends our prior work on backpropagation-based decoding ( Yang et al, 2020 ) using LSTMs for language pairs. Instead, we adapt transformer-based decoders for language triplets and beyond.…”

“…Since English captions and Japanese captions are created independently, the German captions translated from English captions are also independent from Japanese captions. Besides machine translation, we can also get better image captions for target languages as conducted in Yang et al (2020) by conditioning on the source language and a given input image I . The only difference is during validation and test time, where γ is initialized as γ = f 1 ( I ) instead of using random noise as input to f 1 .…”

“…In the past few years, there have been several efforts in taking advantage of images to discover and enhance connections across different languages ( Gella et al, 2017 ; Nakayama and Nishida, 2017 ; Elliott and Kádár, 2017 ). While some works have exploited alignments at the word-level ( Bergsma and Van Durme, 2011 ; Hewitt et al, 2018 ), recent work has moved forward to finding alignments between complex sentences ( Barrault et al, 2018 ; Surís et al, 2020 ; Sigurdsson et al, 2020 ; Yang et al, 2020 ).…”

“…Once this model is pre-trained, we use energy-based decoding that relies on the backpropagation algorithm to use the output from a source language decoder as additional input to generate a sentence in a target language decoder. Backpropagation as a decoding mechanism has been used in some recent language generation work ( Qin et al, 2020 ; Yang et al, 2020 ).…”