Analysis of Positional Encodings for Neural Machine Translation

Rosendahl, Jan; Tran, Viet Anh; Wang, Weiyue; Ney, Hermann

doi:10.5281/zenodo.3525024

Cited by 4 publications

(1 citation statement)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…RPE outperforms APE on out-of-distribution data in terms of sequence length owing to its innate shift invariance (Rosendahl et al, 2019;Neishi and Yoshinaga, 2019;Narang et al, 2021;Wang et al, 2021). However, the self-attention mechanism of RPE involves more computation than that of APE 4 .…”

Section: Relative Position Embedding (Rpe)mentioning

confidence: 99%

SHAPE: Shifted Absolute Position Embedding for Transformers

Kiyono

Kobayashi

Suzuki

et al. 2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Position representation is crucial for building position-aware representations in Transformers. Existing position representations suffer from a lack of generalization to test data with unseen lengths or high computational cost. We investigate shifted absolute position embedding (SHAPE) to address both issues. The basic idea of SHAPE is to achieve shift invariance, which is a key property of recent successful position representations, by randomly shifting absolute positions during training. We demonstrate that SHAPE is empirically comparable to its counterpart while being simpler and faster 1 .

show abstract

Section: Relative Position Embedding (Rpe)mentioning

confidence: 99%

SHAPE: Shifted Absolute Position Embedding for Transformers

Kiyono

Kobayashi

Suzuki

et al. 2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

show abstract

A New Training Pipeline for an Improved Neural Transducer

Zeyer

Merboldt²,

Schlüter

et al. 2020

Interspeech 2020

View full text Add to dashboard Cite

The RNN transducer is a promising end-to-end model candidate. We compare the original training criterion with the full marginalization over all alignments, to the commonly used maximum approximation, which simplifies, improves and speeds up our training. We also generalize from the original neural network model and study more powerful models, made possible due to the maximum approximation. We further generalize the output label topology to cover RNN-T, RNA and CTC. We perform several studies among all these aspects, including a study on the effect of external alignments. We find that the transducer model generalizes much better on longer sequences than the attention model. Our final transducer model outperforms our attention model on Switchboard 300h by over 6% relative WER.

show abstract