Input Combination Strategies for Multi-Source Transformer Decoder

Abstract

In multi-source sequence-to-sequence tasks, the attention mechanism can be modeled in several ways. This topic has been thoroughly studied on recurrent architectures. In this paper, we extend the previous work to the encoder-decoder attention in the Transformer architecture. We propose four different input combination strategies for the encoderdecoder attention: serial, parallel, flat, and hierarchical. We evaluate our methods on tasks of multimodal translation and translation with multiple source languages. The experiments show that the models are able to use multiple sources and improve over single source baselines.

Multimodal machine translation through visuals and speech

“…Follow-up studies extend the decoder-based visual attention approach in different ways: Calixto et al (2017) reimplement the gating mechanism (Xu et al 2015) to rescale the magnitude of the visual information before the fusion, while Libovický and Helcl (2017) introduce the hierarchical attention which replaces the concatenative fusion with a new attention layer that dynamically weighs the modality-specific context vectors. Finally, Arslan et al (2018) and Libovický et al (2018) introduce the same idea into the Transformer-based (Vaswani et al 2017) architectures. Besides revisiting the hierarchical attention, Libovický et al (2018) also introduce parallel and serial variants.…”

Multi-Source Transformer for Kazakh-Russian-English Neural Machine Translation

“…We also experimented with a hierarchical attention mechanism along the lines of Libovický and Helcl (2017) and Libovický et al (2018), but as this did not outperform the simpler combination mechanism in (5) in internal testing, our submitted systems utilized the latter.…”

“…3. Training a multi-encoder (Libovický and Helcl, 2017;Libovický et al, 2018) Transformer system (Vaswani et al, 2017) from…”

“…5).R d model ×d k and W O i (s) ∈ R d k ×d model are trainableparameter matrices which project the key, query and value into a smaller dimensionality. Together withd k = d model /h, we have C (s) ∈ R n×d model .Next, we combine the outputs from the different encoders with a simple projection and sum, similar to whatLibovický et al (2018) refer to as "parallel":C…”

The Universitat d’Alacant Submissions to the English-to-Kazakh News Translation Task at WMT 2019

“…In order to prevent that the errors in the Apertium translation are propagated to the output, the decoder should focus mostly on the SL input. However, according to the analysis of attention carried out by Libovickỳ et al (2018), in the serial multisource architecture of Marian the output seems to be built with information from all inputs. We plan to explore more multi-source architectures in the future.…”