On the Relation between Position Information and Sentence Length in Neural Machine Translation

Abstract

Long sentences have been one of the major challenges in neural machine translation (NMT). Although some approaches such as the attention mechanism have partially remedied the problem, we found that the current standard NMT model, Transformer, has difficulty in translating long sentences compared to the former standard, Recurrent Neural Network (RNN)-based model. One of the key differences of these NMT models is how the model handles position information which is essential to process sequential data. In this study, we focus on the position information type of NMT models, and hypothesize that relative position is better than absolute position. To examine the hypothesis, we propose RNN-Transformer which replaces positional encoding layer of Transformer by RNN, and then compare RNN-based model and four variants of Transformer. Experiments on ASPEC English-to-Japanese and WMT2014 Englishto-German translation tasks demonstrate that relative position helps translating sentences longer than those in the training data. Further experiments on length-controlled training data reveal that absolute position actually causes overfitting to the sentence length.

Sentence Concatenation Approach to Data Augmentation for Neural Machine Translation

“…They stated that this degradation in quality was caused by the short length of the translations. Additionally, Neishi and Yoshinaga (2019) propose to use the relative position information instead of the absolute position information to mitigate the performance drop of NMT models for long sentences. They conducted an analysis of the translation quality and sentence length on lengthcontrolled English-to-Japanese parallel data and showed that the absolute positional information sharply drops the BLEU score of the transformer model (Vaswani et al, 2017) in translating sentences that are longer than those in the training data.…”

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop

“…They stated that this degradation in quality was caused by the short length of the translations. Additionally, Neishi and Yoshinaga (2019) propose to use the relative position information instead of the absolute position information to mitigate the performance drop of NMT models for long sentences. They conducted an analysis of the translation quality and sentence length on lengthcontrolled English-to-Japanese parallel data and showed that the absolute positional information sharply drops the BLEU score of the transformer model in translating sentences that are longer than those in the training data.…”

SHAPE: Shifted Absolute Position Embedding for Transformers

“…RPE outperforms APE on out-of-distribution data in terms of sequence length owing to its innate shift invariance (Rosendahl et al, 2019;Neishi and Yoshinaga, 2019;Narang et al, 2021;Wang et al, 2021). However, the self-attention mechanism of RPE involves more computation than that of APE 4 .…”

“…RPE outperforms APE on sequence-to-sequence tasks (Narang et al, 2021;Neishi and Yoshinaga, 2019) due to extrapolation, i.e., the ability to generalize to sequences that are longer than those observed during training (Newman et al, 2020). Wang et al (2021) reported that one of the key properties contributing to RPE's superior performance is shift invariance 2 , the property of a function to not change its output even if its input is shifted.…”