Document-Level Neural Machine Translation with Hierarchical Attention Networks

Abstract

Neural Machine Translation (NMT) can be improved by including document-level contextual information. For this purpose, we propose a hierarchical attention model to capture the context in a structured and dynamic manner. The model is integrated in the original NMT architecture as another level of abstraction, conditioning on the NMT model's own previous hidden states. Experiments show that hierarchical attention significantly improves the BLEU score over a strong NMT baseline with the state-of-the-art in context-aware methods, and that both the encoder and decoder benefit from context in complementary ways.

Selective Attention for Context-aware Neural Machine Translation

“…Experiments show that our approach improves upon the Transformer by an overall +1.34, +2.06 and +1.18 BLEU for TED Talks, News-Commentary and Europarl, respectively. It also outperforms two recent context-aware baselines Miculicich et al, 2018) in majority of the cases.…”

“…Training The document-conditioned NMT model P θ (y j |x j , D −j ) is realised using a neural architecture and usually trained via a two-step procedure Miculicich et al, 2018). The first step involves pre-training a standard sentence-level NMT model, and the second step involves optimising the parameters of the whole model, i.e., both the document-level and the sentence-level parameters.…”

Enhancing Context Modeling with a Query-Guided Capsule Network for Document-level Translation

“…Main Results Table 2 shows that our model surpasses all the context-agnostic (Vaswani et al, 2017) and context-aware (Zhang et al, 2018a;Miculicich et al, 2018;Maruf and Haffari, 2018) baselines on TED and Europarl datasets. For TED dataset, the performance of our model greatly exceeds that of all other baselines, and is better than Miculicich et al (2018) with a gain of +0.59 BLEU and +0.61 Meteor.…”

“…Main Results Table 2 shows that our model surpasses all the context-agnostic (Vaswani et al, 2017) and context-aware (Zhang et al, 2018a;Miculicich et al, 2018;Maruf and Haffari, 2018) baselines on TED and Europarl datasets. For TED dataset, the performance of our model greatly exceeds that of all other baselines, and is better than Miculicich et al (2018) with a gain of +0.59 BLEU and +0.61 Meteor. For Europarl dataset, our model got improvements with a gain of +0.07 on BLEU metric, but the Meteor score is +0.64 higher than Maruf et al (2019) which utilize the whole document as the contextual information, whereas we only using 3 previous sentences.…”

“…We use the Transformer architecture as our context-agnostic baseline and adopt three contextaware baselines (Zhang et al, 2018a;Miculicich et al, 2018;Maruf and Haffari, 2018).…”